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EXECUTIVE  SUMMARY 


The  current  Department  of  the  Navy  (DoN)  Systems  Engineering  process  has 
documented  instances  of  programs  failing  to  detect  critical  inter-operational  failures  prior  to 
Initial  Operational  Test  &  Evaluation  (IOT&E).  “Approximately  50%  of  programs  completing 
IOT&E  since  2000  have  been  assessed  as  not  operationally  effective  and/or  suitable”  [Defense 
Science  Board  Task  Force,  2008].  Early  detection  of  problems  is  critical  to  prevent  cost 
overruns  and  schedule  delays.  The  cost  and  time  required  to  correct  a  problem  are  directly 
related  to  how  late  in  the  process  the  problem  is  found  and  these  cost  and  schedule  impacts 
increase  rapidly  as  the  system  nears  deployment.  Programs  are  successfully  passing 
developmental  testing;  however,  these  tests  do  not  provide  adequate  assurances  that  the  system 
will  satisfy  user  needs  or  Concept  of  Operations  (CONOPS)  based  interoperating  requirements 
of  the  operational  community.  Furthermore,  requirements  that  are  critical  to  mission  success  are 
not  being  identified  in  the  current  SE  process,  and  allocations  or  dependency  on  supporting 
resources  (systems,  sensors,  data,  etc.)  are  not  being  reflected  in  the  Systems  Engineering, 
contractor  development  plans,  and  testing  plans. 

The  goal  of  this  project  was  to  analyze  the  current  Systems  Engineering  process 
supporting  Department  of  Navy  (DoN)  acquisition  to  increase  the  percentage  of  programs  that 
successfully  pass  Operational  Test  &  Evaluation  (OT&E)  and  therefore  decrease  the  cost  and 
schedule  delays  associated  with  programs  failing  OT&E.  To  do  this,  the  authors  developed  an 
approach  to  track  Interoperability  Readiness  Levels  (EORLs).  As  part  of  this  task,  the  authors 
analyzed  deficiencies  of  the  current  Systems  Engineering  process  supporting  Department  of 
Navy  (DoN)  acquisition  in  the  area  of  interoperability. 

The  deliverables  of  this  project  are  a  detailed  description  of  the  EORL  Process,  the 
results  of  the  table  top  exercise  and  model  validation  used  to  detennine  the  effect  these  changes 
will  have  on  the  current  Systems  Engineering  process,  and  an  assessment  of  their  impact  with 
regard  to  performance  and  cost. 

During  the  process  review  and  problem  identification  phase  the  stakeholder  needs  and 
requirements  were  collected  and  prioritized.  The  current  DoD  Systems  Engineering  process  and 
Operational  Testing  process  were  reviewed  and  evaluated  to  detennine  how  system  level 


requirements  are  developed  from  user  needs  and  subsequently  tested.  Finally,  recently 
developed  systems  that  had  issues  arise  during  OT&E  were  identified  and  examined  to  determine 
potential  sources  of  failure.  The  authors  concentrated  on  areas  in  the  processes  that  may  provide 
insight  into  why  systems  were  initially  passing  DT,  and  then  subsequently  failing  OT&E. 

Based  on  the  results  of  the  open  literature  and  program  documentation  research,  several 
general  conclusions  could  be  made  regarding  the  causes  of  OT&E  failures.  The  bulk  of  the 
systems  cited  in  the  literature  and  the  specific  programs  that  were  reviewed  have  seen  failures  in 
OT&E  due  to  a  lack  of  operational  suitability  or  interoperability.  Additionally,  the  inclusion  of 
OT&E  personnel  early  in  the  development  process  is  necessary  to  ensure  adequate,  realistic 
testing  or  simulation  is  performed  prior  to  OT&E  to  discover  deficiencies.  In  some  cases, 
deficiencies  found  during  early  testing  or  simulations  have  not  been  identified  in  the  Systems 
Engineering  Technical  Review  (SETR)  process.  Or  if  they  have,  programs  somehow  are 
proceeding  to  OT  regardless. 

The  analysis  of  alternatives  phase  consisted  of  concept  development  and  down  select. 
During  this  phase  it  became  apparent  that  the  preferred  solution  of  "Develop  Interoperability 
Readiness  Levels"  implies  one  activity  only,  yet,  this  process  involves  much  more  than  simply 
developing  EORLs.  It  uses  Interoperability  Readiness  Levels  as  a  mechanism  to  assess  the 
maturity  of  the  design  of  a  system  with  regard  to  how  well  it  interoperates  with  other  systems  in 
support  of  mission  critical  threads,  and  as  a  means  to  predict  a  system's  ability  to  pass 
operational  testing  with  regard  to  interoperability. 

In  the  third  phase,  the  detailed  design  was  developed.  To  assess  interoperability 
readiness  levels,  it  was  determined  that  I/ORLs  would  need  to  be  evaluated  at  multiple  SETRs. 
There  is  a  top  level  process  presented  in  developing  I/ORLs  that  considers  how  this  process  will 
be  implemented  at  specific  SETRs.  The  authors  created  an  I/ORL  standard  outlining  the  I/ORL 
levels,  values,  and  descriptions.  EORLs  are  intended  to  assess  the  state  of  a  system’s 
interoperability  using  the  Systems  Engineering  process  within  the  existing  DoD  framework.  The 
I/ORL  scale  ranges  from  1  to  9.  A  value  of  1  is  considered  the  least  mature  as  it  pertains  to 
interoperability,  and  a  value  of  9  is  the  most  mature.  For  each  applicable  review,  an  objective 
I/ORL  value  and  a  threshold  value  was  assigned.  The  objective  value  is  the  level  that  should  be 
met  at  a  particular  review.  A  threshold  value  is  the  minimum  value  that  the  interface  must  meet 


at  the  review.  If  this  value  is  not  met,  the  system  cannot  proceed  to  the  next  technical  review 
without  further  action. 

In  the  event  that  the  I/ORL  review  yields  results  below  the  objective  value,  the  program 
manager  will  need  to  demonstrate  a  risk  mitigation  plan  to  correct  the  issues  prior  to  moving  to 
the  next  phase  of  the  system  development.  The  purpose  of  this  is  to  address  potential  trouble 
areas  in  more  detail  before  they  delay  the  system.  This  approach  highlights  potential  system 
interoperability  issues  allowing  them  to  be  resolved  earlier  in  the  system  development.  While 
this  process  adds  an  administrative  burden,  the  benefits  of  tracking  I/ORLs  and  developing 
mitigation  strategies  outweighs  the  cost  of  this  burden. 

The  process  validation  phase  utilized  two  methodologies.  A  tabletop  exercise  (TTX)  was 
used  to  demonstrate  the  functional  details  of  the  I/ORL  Process  as  it  tracks  the  interoperability  of 
a  fictitious  system.  Secondly,  a  computer-based  model  was  developed  to  track  the  effectiveness 
of  the  I/ORL  Process  if  applied  across  the  Navy/DoD. 

During  the  tabletop  exercise,  the  authors  noted  shortcomings  in  the  I/ORL  Process, 
generated  solutions,  and  applied  improvements  to  the  I/ORL  Process.  The  fictitious  Advanced 
Combat  Integrated  Helmet  (ACIH)  was  selected  as  a  sample  system  for  the  TTX  because  it  has 
many  internal  and  external  interfaces  used  for  communication.  The  audio  and  visual 
communication  between  headquarters  and  ground  troops  was  selected  to  focus  the  TTX  on  the 
ACIH  communications  mission  thread. 

The  second  part  of  the  validation  was  a  computer-based  model  designed  to  simulate  the 
I/ORL  system.  It  simulates  the  interoperability  requirements  progress  throughout  the  System 
Engineering  process,  assigns  I/ORL  values,  and  compares  the  assigned  value  to  the  required 
values  at  each  review.  If  the  system  does  not  meet  the  prescribed  I/ORL  threshold  for  the 
review,  the  system  is  reworked  to  meet  the  threshold  at  an  additional  cost  that  is  assessed 
according  to  the  development  phase. 

Based  on  the  modeling  and  the  table  top  exercise,  the  authors  believe  that  the  current 
Systems  Engineering  process  could  benefit  from  the  implementation  of  I/ORLs.  The  modeling 
estimates  a  savings  of  approximately  14.2%  or  a  reduction  of  6.29%  in  rework  costs  over  the 
current  DoD  System  Engineering  process.  Additionally,  the  model  shows  an  OT&E  pass  rate  of 
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93.73%  with  the  implementation  of  the  I/ORL  evaluations.  This  is  a  significant  increase  in  the 
pass  rate  of  systems  from  DT  to  OT&E. 
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I.  INTRODUCTION 


A.  BACKGROUND 

Department  of  the  Navy  (DoN)  Systems  Engineering  (SE)  processes  and  practices  have 
evolved  over  time.  Systems  Engineering  methods  and  process  guidelines  have  been  in 
development  since  the  early  1960’s.  The  Department  of  Defense  pioneered  these  processes  and 
practices  and  developed  MIL-STD-499  which  was  the  original  specification  for  Systems 
Engineering.  These  Systems  Engineering  methods  have  continued  to  evolve  both  within  the 
federal  government  and  in  private  industry,  particularly  since  the  role  of  military  specifications 
has  diminished  due  to  acquisition  reform.  Current  Systems  Engineering  methods  are 
documented  in  numerous  publications  such  as  the  International  Council  on  Systems  Engineering 
(INCOSE)  Systems  Engineering  Handbook,  and  ISO/IEC  15288  Systems  and  Software 
Engineering  -  System  Life  Cycle  Process.  The  Federal  Government,  and  in  particular  the 
Department  of  the  Navy,  also  specifies  how  these  Systems  Engineering  methods  will  be 
implemented  within  the  context  of  naval  systems  acquisition  and  these  guidelines  are  specified  in 
documents  such  as  the  NAVSEAINST  5000.9  Naval  SYSCOM  Systems  Engineering  Policy, 
Naval  SYSCOM  Engineering  Technical  Review  Handbook.  Although  the  DoN  has  put  in  place 
numerous  system  design  review  checks,  these  enforcement  measures  on  Systems  Engineering 
development  practices  are  not  preventing  systems  from  failing  operational  criteria  associated 
with  Operational  Test  (OT)  at  the  end  of  their  system  development  cycles. 

Independent  Operational  Testing  organizations  can  trace  their  origins  to  the  1970  Blue 
Ribbon  Defense  Panel  Report.  The  report  concluded  that  “There  has  been,  and  is  currently  no 
effective  means  for  conducting  productive  joint  operations  test  and  evaluations.  The  fact  that 
some  such  efforts  heretofore  have  encountered  difficulties  and  achieved  few  useful  results  does 
obviate  the  requirements  for  much  needed  joint  operational  test  and  evaluation  (OT&E)”  [JT&E 
Handbook,  1996].  The  current  role  of  the  operational  test  organizations  is  to  make  an 
independent  detennination  of  the  operational  effectiveness  and  suitability  of  programs  of  record 
prior  to  full  rate  production  and  deployment.  Prior  to  the  establishment  of  the  independent 
OT&E  organizations,  individual  commands  were  responsible  for  all  testing  prior  to  delivery  of 
systems  to  the  field;  which  led  to  many  problems  after  delivery.  Individual  commands, 
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specifically  the  program  offices,  retain  cognizance  over  Developmental  Test  and  Evaluation 
(DT&E). 

B.  PROBLEM  STATEMENT 

The  current  Department  of  the  Navy  (DoN)  Systems  Engineering  process  has 
documented  instances  of  programs  failing  to  detect  critical  inter-operational  failures  prior  to 
Initial  Operational  Test  &  Evaluation  (IOT&E).  “Approximately  50%  of  programs  completing 
IOT&E  since  2000  have  been  assessed  as  not  operationally  effective  and/or  suitable”  [Defense 
Science  Board  Task  Force,  2008].  Early  detection  of  problems  is  critical  to  prevent  cost 
overruns  and  schedule  delays.  The  cost  and  time  required  to  correct  a  problem  are  directly 
related  to  how  late  in  the  process  the  problem  is  found  and  these  cost  and  schedule  impacts 
increase  rapidly  as  the  system  nears  deployment.  Programs  are  successfully  passing 
developmental  testing;  however,  these  tests  do  not  provide  adequate  assurances  that  the  system 
will  satisfy  user  needs  or  Concept  of  Operations  (CONOPS)  based  interoperating  requirements 
of  the  operational  community.  Furthermore,  requirements  that  are  critical  to  mission  success  are 
not  being  identified  in  the  current  SE  process,  and  allocations  or  dependency  on  supporting 
resources  (systems,  sensors,  data,  etc.)  are  not  being  reflected  in  the  Systems  Engineering, 
contractor  development  plans,  and  testing  plans. 

Department  of  Defense  Instruction  (DoDI)  5000.02  and  related  documents  describe  the 
Defense  Acquisition  System  used  for  development  of  Department  of  the  Navy  programs. 

Despite  extensive  guidance  and  multiple  decision  gates,  programs  are  still  failing  to  ensure  that 
all  requirements  critical  to  meeting  the  user  need  are  identified  and  properly  tested  prior  to  the 
start  of  the  OT&E  process.  Improvements  must  also  be  made  to  the  Systems  Engineering 
requirements  process  to  ensure  that  these  critical  requirements  are  made  visible  at  key  technical 
reviews(s)  such  as  Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR),  Test 
Readiness  Review  (TRR),  and  Production  Readiness  Review  (PRR). 

C.  RESEARCH  QUESTIONS 

The  following  research  questions  guided  the  direction  of  this  project’s  research  and 
analysis  and  were  answered  as  the  work  progressed: 

•  How  are  mission  critical  elements  identified  and  managed  using  the  existing  acquisition 

and  SE  processes? 
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•  What  are  the  common  failures  in  the  engineering  process  that  result  in  missing  mission 
critical  elements?  Where  do  these  failures  occur? 

•  How  prevalent  are  mission  critical  failures  of  programs  discovered  during  OT&E? 

•  What  kinds  of  Modeling  and  Simulation  (M&S)  can  be  leveraged  to  analyze  DoN 
Systems  Engineering  processes? 

•  Can  process  improvements  be  used  to  supplement  the  DoN  Systems  Engineering 
processes  to  improve  handling  of  mission  critical  interoperability  elements  of  programs? 

•  What  are  the  cost  ramifications  and  possible  benefits  of  implementing  these  enhanced 
processes? 

D.  INITIAL  ASSUMPTIONS 

It  is  understood  that  Systems  Engineering  involves  balancing  the  often  conflicting 
requirements  of  development  costs,  development  schedules  and  system  performance.  However, 
this  a  significant  oversimplification  of  the  situation  due  to  the  interrelations  between  cost  and 
schedule  that  exist  in  the  budgetary  process.  While  on  the  surface,  making  small  investments  in 
time  and  money  early  in  the  process  in  order  to  save  much  larger  amounts  of  money  in  the  later 
lifecycles  would  seem  like  an  obvious  choice;  this  is  actually  quite  difficult  due  to  how  funding 
is  allocated  into  the  different  “colors”  of  money  and  how  this  funding  is  time  phased  through  the 
program.  The  authors  do  not  address  these  issues  in  this  paper.  The  discussions  in  this  paper 
regarding  how  money  should  be  allocated  to  various  types  of  testing  and  when  this  testing  should 
occur  do  not  take  into  account  the  reprogramming  of  funding  that  must  occur  to  facilitate  these 
changes  or  the  levels  of  approval  that  these  changes  would  require. 

E.  PROJECT  GOALS  AND  DELIVERABLES 

The  goal  of  the  project  is  to  analyze  the  deficiencies  of  the  current  Systems  Engineering 
process  supporting  Department  of  Navy  (DoN)  Acquisition  in  the  area  of  requirements  critical  to 
ensuring  mission  success.  This  includes  detailing  recommended  changes  to  the  Systems 
Engineering  (SE)  processes  in  the  Defense  Acquisition  System  in  order  to  increase  the 
percentage  of  programs  that  successfully  pass  Operational  Test  &  Evaluation  (OT&E)  and 
therefore  decrease  the  cost  and  schedule  delays  associated  with  programs  failing  OT&E.  The 
deliverables  of  this  project  are  a  detailed  description  of  the  recommended  process  changes,  the 
results  of  the  validation  methods  used  to  detennine  the  effect  these  changes  will  have  on  the 
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current  SE  process,  and  an  assessment  of  their  impact  with  regard  to  performance  and  cost.  It 
should  be  noted  that  during  the  initial  research  phase  of  this  project  the  scope  was  reduced  to 
only  address  deficiencies  related  to  interoperability,  and  the  deliverables  listed  above  reflect  the 
narrowed  solution  space. 

F.  TAILORED  SYSTEMS  ENGINEERING  PROCESS 

The  authors  implemented  a  tailored  Systems  Engineering  approach  that  started  with  the 
identification  of  the  customers’  needs  and  proceeded  through  the  phases  illustrated  in  Figure  1 
until  a  recommended  solution  was  generated.  This  approach  was  based  on  the  waterfall  model 
introduced  by  Royce  [Blanchard,  2006]  and  has  been  tailored  to  contain  elements  from  Buede 
[Buede,  2009]  and  lifecycle  design  to  specifically  address  the  goals  of  the  project.  This 
customized  process  focuses  on  the  development  of  the  system  and  less  on  its  support,  given  its 
ideological  nature. 

The  project  consisted  of  five  phases:  Process  Review  and  Problem  Identification, 
Analysis  of  Alternatives,  Detailed  Process  Design,  Process  Validation,  and  Present 
Recommendations  as  shown  in  Figure  1.  Each  phase  was  executed  by  taking  the  inputs  and 
feedback  from  later  phases.  Once  the  given  outputs  were  of  suitable  quality,  then  each  phase 
was  considered  complete.  A  detailed  description  of  each  phase  is  found  in  the  subsequent 
sections. 
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Figure  1.  Tailored  SE  Process 

This  Systems  Engineering  process  was  developed  for  studying  this  problem  and  development  of  the  solution. 


1.  Process  Review  and  Problem  Identification 

All  stakeholder  needs  and  requirements  were  collected  and  prioritized  based  on 
pertinence  to  the  project  objectives  and  stakeholder  importance.  Conflicting  needs  and 
requirements  were  discussed  with  the  required  stakeholders  to  reach  an  agreement  before  the 
authors  baselined  the  requirements. 

The  team  reviewed  the  current  guidance  and  instructions  on  applying  Systems 
Engineering  in  Department  of  Defense  (DoD)  acquisitions.  The  goal  was  to  examine  the 
process’s  overall  effectiveness  at  identifying  the  mission  critical  requirements  and  evaluating 
these  requirements  during  developmental  testing.  The  process  was  evaluated  to  determine  how 
system  level  requirements  are  developed  from  user  needs.  Further  investigation  was  conducted 
to  examine  how  operational  tests  are  developed  based  upon  the  system  requirements  and  the 
underlying  user  needs. 

Research  was  done  to  identify  recently  developed  systems  that  passed  and  failed 
operational  testing.  Each  system  was  examined  to  determine  sources  of  success  and  causes  of 
failure.  This  was  accomplished  by  addressing  questions  such  as: 
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•  How  are  system  requirements  captured  in  a  Capability  Development  Document  (CDD) 

supported  by  (or  connected  to)  doctrinally-defined  missions? 

•  Are  appropriate  user-centric,  solution-neutral  metrics  identified? 

•  Are  a  system’s  external  interfaces  identified? 

•  Are  realistic  Concepts  of  Operation  (CONOPS)  used? 

•  Were  testers  and  end  users  formally  involved  in  the  Systems  Engineering  Technical 

Review  (SETR)? 

The  system’s  ability  to  meet  their  requirements  and  objectives  that  satisfy  the 
stakeholders’  needs  were  analyzed  to  determine  the  mission  critical  beneficial  concepts  and 
shortfalls  for  a  system’s  engineering  process. 

2.  Generation  and  Analysis  of  Alternatives 

The  analysis  of  alternatives  or  down-select  was  accomplished  following  the  identification 
of  the  source  of  failures  pertaining  to  requirements  critical  to  mission  success  within  the  Systems 
Engineering  process.  Key  decisive  criteria  were  established  at  this  phase  to  detennine  feasible 
alternatives.  The  application  of  decisive  criteria  was  employed  to  exclude  undesirable  solutions 
and  highlight  acceptable  alternatives.  Multiple  process  concepts  were  developed  based  upon 
these  decisive  criteria.  Each  concept  was  evaluated  for  the  relative  cost  of  implementation  and 
the  cost  of  execution.  These  concepts  were  also  rated  based  on  the  quality  of  process 
improvement,  looking  at  factors  concerning  thoroughness,  traceability,  and  simplicity.  The 
resulting  concepts  were  evaluated  and  chosen  to  develop  the  best  mission  criticality  process 
based  on  cost-effectiveness  and  process  improvement  in  order  to  prevent  the  shortfalls  identified 
in  the  previous  phase. 

The  programs  identified  in  the  previous  phase  were  evaluated  to  detennine  what  program 
was  going  to  be  used  as  the  representative  example.  This  evaluation  was  done  based  on 
availability  of  critical  information,  team  knowledge  of  the  programs,  and  the  representative 
nature  of  each  program.  The  representative  nature  was  defined  as  being  of  sufficient  complexity 
to  adequately  demonstrate  the  process  and  also  the  likelihood  of  the  problems  encountered  on 
that  program. 
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3.  Detailed  Process  Design 

The  selected  process  concept  was  used  to  develop  a  mission  criticality  process  that  can  be 
integrated  into  the  DoD  Systems  Engineering  process.  The  new  process  included  new 
deliverables  such  as:  new  exit/entrance  criteria,  modified  documents,  new/modified  instructions, 
and  organizational  charts  that  describe  roles  and  responsibility.  Mission  Criticality  Process 
Alternatives  were  used  as  reference  material  when  encountering  detail  design  issues.  Concepts 
for  improvement  were  taken  from  the  alternatives  developed  during  the  AoA  phase  during  the 
detailed  design.  While  designing  the  new  process,  the  mission  criticality  validation  criteria  were 
developed.  In  the  next  phase,  this  process  was  simulated  via  a  model  on  the  representative 
program  selected  in  this  phase. 

4.  Process  Validation 

The  new  mission  criticality  process  was  evaluated  on  the  representative  system  by 
perfonning  table-top  exercises.  The  exercises  used  the  validation  criteria  developed  during  the 
detailed  process  design  phase  to  evaluate  the  process.  Cost  data  concerning  the  execution  of  the 
process  was  collected  while  the  process/model  was  validated.  The  results  from  the  evaluation 
were  analyzed  and  reported. 

5.  Present  Recommendations 

Once  the  process  was  simulated  and  analyzed,  the  findings  and  recommendations  were 
presented  to  the  stakeholders  and  are  reported  in  this  document.  This  report  includes  how 
successful  the  new  process  was  at  addressing  the  problems  encountered  on  the  representative 
system  and  how  other  systems  examined  could  have  benefited  from  this  new  process.  This 
report  includes  a  cost  analysis  for  implementation  of  this  process  for  future  systems. 
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II.  PROCESS  REVIEW  AND  PROBLEM  REFINEMENT 


The  authors  accomplished  several  tasks  concurrently  during  this  initial  phase  of  the 
project.  First,  stakeholder  needs  and  requirements  were  collected  and  prioritized.  Second,  the 
current  DoD  Systems  Engineering  process  and  Operational  Testing  process  were  reviewed  and 
evaluated  to  determine  how  system  level  requirements  are  developed  from  user  needs  and  tested. 
Finally,  recently  developed  systems  that  had  issues  arise  during  OT  were  identified  and 
examined  to  determine  potential  sources  of  failure.  The  results  of  these  tasks  are  discussed  in 
more  detail  in  the  paragraphs  below. 

A.  STAKEHOLDER  NEEDS  ANALYSIS 

The  stakeholders’  needs  for  this  project  are  based  on  the  results  of  literature  research, 
communication  with  the  stakeholders,  and  team  experience.  The  stakeholder  needs  translate  into 
derived  requirements  that  capture  the  system’s  capabilities  and  constraints  for  perfonnance 
design  parameters.  It  was  important  to  the  team  to  understand  the  stakeholders’  needs,  as  well  as 
the  mission  requirements,  to  transform  them  into  defined  requirements  for  the  Systems 
Engineering  detailed  design  phase  of  the  project  [Buede,  2009]. 

1.  Stakeholder  Identification 

The  authors  produced  a  list  of  potential  stakeholders  for  the  project.  This  list  was  further 
synthesized  to  identify  additional  candidates  to  ensure  that  each  had  an  interest  in  the  results  of  a 
design  solution.  Also  important  to  this  identification  process,  was  the  distinction  of  which 
stakeholders  may  be  impacted  as  a  result  of  the  project  findings  and  recommendations.  As  a 
result  of  this  process,  the  following  stakeholders  were  identified: 

•  ASN  (RDA)  CHSENG 

The  Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition  Chief 
Systems  Engineer’s  Office  is  responsible  for  Systems  Engineering  Technical  Reviews  (SETR) 
for  all  AC  AT  programs,  regardless  of  the  AC  AT  designation  [ASN(RDA),  2008],  CHSENG 
staffers  are  interested  in  the  utilization  of  the  possible  process  changes  for  incorporation  into  the 
SETR  [SETR,  2009]  process.  ASN  (RDA)  CHSENG  also  served  as  the  sponsor  for  this  project, 
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the  authors  were  engaged  with  his  representative  on  a  continuous  basis  to  understand  and  receive 
guidance  on  the  direction  of  the  project. 

•  Program  Office/Pro  gram  Managers 

The  Program  Offices  and  Program  Managers  for  specific  development  systems  may  be 
impacted  by  this  project.  The  programs  may  be  able  to  utilize  the  output  of  this  project  to  re¬ 
define  their  Systems  Engineering  process,  requirements  definition,  test  planning  and  conduct. 

•  Naval  War  Fighter 

The  key  stakeholder  for  this  project  is  the  naval  war  fighter  or  “user”  of  the  system. 
Although  the  war  fighter  does  not  encounter  the  identified  problem  until  a  system  becomes 
operational,  the  war  fighter  is  left  to  deal  with  consequences  of  an  ineffective  system  that  is 
delivered  with  deficiencies.  The  Joint  Capabilities  Integration  and  Development  System 
(JCIDS)  [CJCSI  3170.01G,  2009]  process  describes  the  Joint  Forces  Command  (JFCOM)  as  the 
war  fighter  representative  into  the  acquisition  infrastructure.  This  representation  is  flowed  down 
from  the  Joint  Requirements  Oversight  Council  (JROC)  to  the  user.  The  war  fighter  or  “user”  is 
defined  as:  “An  operational  command  or  agency  that  receives  or  will  receive  benefit  from  the 
acquired  system”  [CJCSI  3 170.0 1G,  2009]. 

•  System  Design,  Development  and  Validation  Teams 

The  design  engineering  teams  may  be  affected  by  the  outcome  of  this  project.  This  could 
be  the  prime  contractors,  working  with  the  Program  Office  and  also  the  System  Commands 
(SYSCOM)  field  activities  charged  with  the  system  development  efforts.  The  criteria  identified 
for  “mission  criticality”  may  have  impacts  on  the  design  and  development  of  a  system,  along 
with  the  testing  requirements  identified. 

•  U.S.  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

OPTEVFOR  is  the  independent  operational  test  agent  responsible  for  assessing  the 

operational  effectiveness  and  suitability  of  new  and  improved  war  fighting  systems  and 
capabilities  for  the  military  services.  The  U.S.  Navy  has  its  own  specific  evaluators  when 
reviewing  a  Navy  only  system  that  has  no  joint  services  capabilities.  The  outcome  of  this  project 
may  provide  additional  information  in  order  to  evaluate  systems  under  test. 
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2.  Stakeholder  Initial  Requirements 

As  a  result  of  the  stakeholder  identification  process,  perceived  needs  were  documented 
from  the  definition  of  the  initial  problem  statement.  These  perceived  needs  were  further 
analyzed  to  ascertain  which  ones  would  have  an  effect,  if  any,  on  which  stakeholders.  This 
analysis  resulted  in  the  identification  of  requirements  that  would  be  of  interest  to  multiple 
stakeholders.  The  stakeholder  requirements  analysis  provided  the  following: 

A.  Develop  a  Method  to  Track  Mission  Critical  Requirements 

The  first  requirement,  a  method  to  track  mission  critical  requirements  was  derived  from 
the  ASN  (RDA)  Chief  Systems  Engineer  Staffer  project  proposal  discussion  [McKinlay,  2010], 
This  requirement  would  also  be  shared  by  the  program  offices  and  system  design,  development 
and  validation  teams.  The  outcome  for  this  requirement  is  to  capture  and  track  critical  needs 
related  to  a  specific  mission  or  missions. 

B.  Explore  a  Mission  Thread  with  Respect  to  Mission  Critical  Requirements 

As  a  continuation  of  this  need,  the  ASN  (RDA)  Chief  Systems  Engineer  specifically 
requested  the  exploration  of  mission  threads  with  respect  to  mission  critical  requirements  in 
order  to  assess  the  impact  to  successful  completion  of  OT.  The  objective  of  this  is  to 
demonstrate  how  mission  critical  requirements  can  be  derived  from  mission  threads. 

C.  Develop  a  Conceptual  Approach  to  Improve  OT  Results 

The  intent  of  developing  a  conceptual  approach  to  improve  OT  results  is  to  allow  the 
authors  to  present  findings  in  a  methodology  that  could  be  implemented  in  a  system  to  identify 
and  track  mission  critical  requirements. 

D.  Improve  OT  Success  Rate 

An  inherent  need  from  stakeholders  is  that  the  proposed  solution  offers  an  improved  OT 
success  rate  when  compared  to  the  current  process. 

E.  Reduce  Rework  Costs 

Finally,  the  program  offices  and  system  design,  development,  and  validation  teams 
communicated  that  they  would  desire  a  net  reduced  development  cost  as  a  result  of  the 
implementation  of  the  final  recommendation.  This  net  reduced  cost  will  be  realized  by  the 
reduction  in  rework  from  systems  passing  the  first  phase  of  verification  testing  known  as 
Developmental  Testing  (DT),  but  failing  to  pass  the  final  validation  phase  of  testing,  known  as 
Operational  Testing  (OT). 
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The  combined  results  of  the  stakeholder  requirements  analysis  are  provided  in  Table  1. 
Table  1.  Stakeholders  Requirements  Results 

This  table  shows  the  relationship  between  key  stakeholders  and  high  level  requirements  of  this  process. 


^""-^-Stakeholders 

Requirement''''---^ 

ASN(RDA) 

CHSENG 

Program 

Office/ 

Manager 

Naval  War 
Fighter 

Design  & 
Development 
Team 

OPTEVFOR 

A.  Develop  Method  to 
Track  Mission  Critical 
Requirements 

• 

• 

• 

B.  Explore  a  Mission 
Thread  with  Respect 
to  Mission  Critical 
Requirements 

• 

C.  Develop 

Conceptual  Approach 
to  Improve  OT  Results 

• 

• 

• 

D.  Improve  OT 

Success  Rate 

• 

• 

• 

• 

• 

E.  Reduce  Rework 
Costs 

• 

• 

B.  PROCESS  RESEARCH 

The  research  process  for  the  Process  Review  and  Problem  Identification  Phase  of  the 
project  comprised  the  following  research  segments:  Requirements  Process,  Operational  Test 
Process  and  Failed  Systems  research.  Each  segment  required  investigating  and  documenting 
possible  root  causes  for  operational  test  failures.  The  authors  concentrated  on  areas  in  the 
processes  that  may  provide  insight  into  why  system  were  initially  passing  DT,  and  then 
subsequently  failing  OT.  Figure  2  provides  an  illustration  of  the  segments  to  the  process 
research  phase  for  the  project. 
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Process  Research 


Figure  2.  Process  Research 

The  Process  Research  Phase  of  this  project  was  comprised  of  research  into  three  areas  and  was  directed  towards 

determining  the  root  causes  of  operational  test  failures. 


1.  Requirements  Process 

a.  Research  Methodology  and  Objectives 

A  principal  goal  of  the  first  stage  of  the  project  was  to  conduct  research  on  the  overall 
Operation  of  the  Defense  Acquisition  System  [DoDI  5000.02,  Enclosure  12,  2008]  with  respect 
to  how  requirements  are  managed  throughout  the  process.  This  is  much  more  than  merely  the 
initial  determination  of  requirements  but  also  includes  requirements  documentation,  validation, 
decomposition,  and  testing,  etc.  This  research  includes  much  of  the  Systems  Engineering 
process  that  is  embedded  into  the  Defense  Acquisition  System  process.  The  authors  focused 
their  research  efforts  on  the  regulatory  guidance  to  determine  how  the  process  is  “supposed”  to 
work.  It  is  important  to  note  here  that  the  program  office  development  organizations  are 
responsible  for  the  generation  of  the  artifacts  produced  as  a  result  of  the  acquisition  design  and 
development  process.  These  documents  are  required  for  incremental  delivery  to  the  evaluation 
authorities  in  concert  with  the  Systems  Engineering  Technical  Review  Process  [SETR,  2009]. 
The  quality,  clarity  and  completeness  of  this  documentation  are  important  in  the  evaluation  of 
systems  as  it  progress  through  the  acquisition  process. 

The  scope  of  the  process  that  the  authors  studied  for  Process  Review  and  Requirements 
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Identification  Phase  of  this  project  is  shown  in  Figure  3.  This  process  starts  with  the 
determination  of  system  requirements  based  upon  the  needs  identified  by  the  Capability  Based 
Assessment  (CBA)  process  that  occurs  as  part  of  Joint  Capability  and  Integration  Development 
System  (JCIDS)  [CJCSI  3170.01G,  2009]  and  includes  the  overall  process  of  determining  how 
well  the  system  meets  those  requirements  over  the  course  of  the  system  development  process. 
Additionally,  this  process  includes  how  these  requirements  are  communicated  from  the  user 
community  to  the  development  community  and  how  these  requirements  are  documented. 


MDD 

♦ 


Material  Solution 
Analysis 


MSA 

♦ 


Technology  Development 


MS  B 

♦ 


Engineering  &  Manufacturing 
Development 


®  t  [ 

is 


OT&E 
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Concept 
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Develop  Technology 


Assess  Progress  &  Manage  Risk 


DT&E 


Develop 

Concepts 


Develop  Technology 


Develop  /  Produce  Prototypes 


CT&E 


Figure  3.  Overview  of  the  Systems  Development  Process 


This  high  level  overview  of  the  systems  development  process  was  adapted  from  CJCSI  3 17 0.0  IG. 


The  end  goal  of  this  process  is  to  have  confidence  that  when  systems  reach  Milestone  C, 
they  will  be  able  to  provide  the  required  operational  capability  in  the  intended  environment. 

Since  problems  are  being  encountered  during  operational  test,  one  can  assume  that  there  is  a  flaw 
in  the  process  or  that  it  is  not  being  followed  correctly.  The  overall  time  of  interest  is  from  the 
CBA  process  that  is  performed  prior  to  the  Material  Development  Decision  (MDD),  through 
Milestone  C,  which  is  essentially  the  final  gate  prior  to  proceeding  to  Operational  Test  & 
Evaluation  (OT&E).  That  being  said,  official  program  initiation  does  not  begin  until  Milestone 
B  and  the  work  performed  in  the  early  phases  is  to  lay  the  foundation  for  the  formal  Defense 
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Acquisition  Program  that  begins  at  that  point.  The  entire  acquisition  process  varies  depending 
on  the  nature  of  the  program.  For  example,  a  newly  designed  weapons  combat  system  could  take 
over  5  years  to  complete  its  acquisition  cycle,  while  a  system  or  component  already  in  the  fleet 
could  take  as  little  as  6  months  to  integrate  since  it  is  already  developed.  This  acquisition 
timeline  varies  according  to  the  complexity,  technology  and  funding  provided  to  programs. 

Figure  3  is  at  too  high  of  a  level  for  any  amount  of  meaningful  assessment,  so  the  authors 
developed  process  maps  that  were  at  a  lower  level  so  specific  steps  in  the  process  could  be 
identified  for  potential  improvements.  These  improvements  were  viewed  from  the  perspective  of 
possible  changes  that  could  be  made  early  in  the  acquisition  cycle  that  would  improve  the  OT 
pass  criteria.  The  depiction  of  the  processes  in  a  graphical  representation  provides  the  visual 
insight  to  understand  any  inter-dependencies  in  the  processes  and  associations  that  may 
otherwise  not  be  apparent.  The  authors  reviewed  Department  of  Defense,  as  well  as  Department 
of  the  Navy  regulations  and  guides  [CJCSI  3170.01G;  DoDI  5000.02,  2008;  SECNAV  5000.2D, 
2008]  to  identify  the  key  steps  that  must  occur  in  order  to  translate  the  capability  needs  into 
system  requirements  and  then  ensure  that  the  system  is  going  to  meet  those  requirements.  Inputs 
for  these  steps  were  identified  including  the  Initial  Capabilities  Document  (ICD),  Capability 
Development  Document  (CDD),  system  specification,  and  other  requirements  documents  in 
addition  to  additional  sources  of  infonnation  that  are  available  [DAU  Guidebook,  2010;  JCIDS 
Manual,  2009].  Outputs  were  also  identified  and  consist  primarily  of  the  work  products  that  are 
generated,  including  specifications,  architectures,  etc.  The  authors  also  determined  the  key 
individuals  that  are  involved  in  the  SE  process.  These  included  high-level  personnel  that  are 
responsible  for  signing  documents  and  authorizing  actions,  along  with  support  staff  that  perfonn 
formal  or  infonnal  assessments  which  are  used  as  the  basis  for  those  high-level  decisions  [DoDD 
5000.01,  2007;  DoDI  5000.02,  2008].  Finally,  the  authors  conducted  an  initial  assessment  of  the 
regulations  to  locate  potential  weaknesses  based  on  sound  Systems  Engineering  principles  and  to 
isolate  what  steps  in  the  process  had  the  highest  potential  for  improvement. 

b.  Capability  Based  Assessment  Phase 

The  Capability  Based  Assessment  (CBA)  is  the  first  phase  of  the  process  and  is 
conducted  prior  to  the  decision  to  pursue  a  materiel  solution.  The  key  activities  and  work 
products  of  this  process  are  shown  in  Figure  4.  Figures  4  through  7  that  follow  were  adapted 
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from  the  Life  Cycle  chart  produced  by  the  Integrated  Defense  Acquisition,  Technology,  and 
Logistics  Life  Cycle  Management  System,  DAU  Press,  Version  5.3.4  of  15  June  2009  [DAU, 
2009].  The  ICD  captures  the  results  of  the  Capabilities  Based  Assessment  if  a  materiel  solution 
is  recommended.  Alternatively,  a  DOTMLPF  Change  Request  (DCR)  describes  a  non-materiel 
solution  if  one  is  determined  to  be  feasible.  For  Navy  systems,  Chief  of  Naval  Operations 
(CNO)  Code  N8  is  responsible  for  acting  as  the  user  representative  and  oversees  the  CBA 
assessment  once  the  Joint  CONOPS  has  been  developed  by  the  Joint  Staff.  At  this  point  in  the 
process  when  a  materiel  solution  is  required  by  an  approved  ICD,  the  Milestone  Decision 


Authority  (MDA)  determines  the  scope  of  the  subsequent  follow  on  analysis  such  as  the  Analysis 
of  Alternatives  (AoA)  [CJCSI  3170.01G,  2009]. 


DoD 

Strategic 

Guidance 


Work  Products 

CBA  -  Capability  Based  Assessment 
DCR  -  DOTMLPF  Change  Recommendation 
ICD  -  Initial  Capability  Document 
JCD  -  Joint  Capability  Document 


Figure  4.  Capability  Based  Assessment  Process 

The  capability  based  assessment  is  the  initial  JCIDS  process  for  determining  if  a  materiel  or  non-materiel  solution 

is  required  to  address  a  user  need. 

Based  on  the  documents  reviewed,  there  is  a  heavy  emphasis  placed  upon  the 
performance  aspects  of  meeting  the  capability  needs.  Identifications  of  constraints  was 
discussed  in  the  DAU  Guidebook  [DAU,  2010];  however,  it  is  unclear  how  thorough  this 
constraint  identification  process  is  and  the  degree  to  which  the  constraints  are  incorporated  into 
the  AoA  plan.  These  constraints  would  include  the  operational  environment,  the  support 
infrastructure  that  is  expected  to  be  available,  and  what  interfaces  would  be  required  to  external 
systems. 
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c.  Material  Solution  Analysis  Phase 

The  Material  Solution  Analysis  phase  is  built  around  a  traditional  V-based  SE  process, 
starting  with  an  analysis  of  the  system  needs.  The  major  difference  in  this  phase  compared  with 
the  traditional  system  development  V-based  SE  process  is  that  no  real  product  is  produced.  The 
design  ideas  and  concepts  are  “paper-based”  and  are  produced  as  desk-top  analyses  at  this  point. 
This  would  also  include  any  modeling  and  simulation  that  may  be  required.  The  process 
continues  with  development  of  multiple  concepts  down  to  a  level  sufficient  for  a  thorough 
comparison  of  those  concepts,  and  then  testing  of  those  concepts  for  their  relative  ability  to  meet 
the  user  needs  within  the  constraints.  The  key  activities  and  work  products  of  this  process  are 
shown  in  Figure  5. 


Figure  5.  Material  Solution  Analysis  Phase 

The  Material  Solution  Analysis  Phase  selects  a  preferred  concept  for  meeting  the  identified  user  need. 


The  Analysis  of  Alternatives  (AoA)  is  the  major  activity  in  this  phase  and  is  an  analytical 
comparison  based  on  operational  effectiveness,  suitability,  and  life-cycle  cost.  However,  the 
authors’  review  of  the  Defense  Acquisition  Guidebook  [DAU,  2010]  showed  a  clear  bias  towards 
analysis  of  effectiveness  and  life-cycle  cost  but  no  mention  of  how  to  incorporate  operational 
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suitability  into  this  analysis.  A  system  that  is  not  reliable  or  cannot  be  maintained  will  not  be  of 
use  to  the  war-fighter,  even  if  the  theoretical  operational  effectiveness  of  the  system  is  very  high. 
Logistics,  packaging,  handling,  and  other  factors  were  included  in  the  Alternative  Systems 
Review  checklist  but  no  clear  guidance  on  how  much  weight  suitability  factors  should  carry  in 
the  selection  of  the  preferred  concept. 

In  addition,  technical  risk  is  not  mentioned  within  the  context  of  the  AoA.  The  concern 
here  is  that  a  concept  will  be  selected  that  has  very  promising  performance;  however,  the 
estimated  performance  is  based  upon  technology  that  is  not  yet  mature.  If  the  substitution  of 
these  technologies  becomes  necessary  due  to  lack  of  Technology  Readiness  Levels  (TRL) 
growth,  there  is  a  significant  risk  that  the  actual  delivered  performance  of  a  system  will  be  quite 
low  even  though  the  program  initially  offered  very  good  perfonnance. 

Another  area  of  concern  is  the  lack  of  continued  development  of  the  concept  for  how  the 
system  will  be  used.  In  the  CBA  process  prior  to  MDD,  mission  threads  are  developed  that 
describe  how  the  overall  mission  will  be  accomplished  and  how  this  system  fits  into  that  mission 
thread.  However,  at  that  point  in  the  process,  the  specifics  of  the  system  have  not  yet  been 
defined.  At  the  end  of  the  Material  Solution  Analysis  phase,  a  preferred  concept  for  a  system  has 
been  developed.  It  would  now  be  possible  to  specify  additional  details  as  to  how  the  system  will 
operate  within  the  context  of  the  mission  threads  that  were  previously  identified.  The  authors 
found  no  indication  that  a  detailed  concept  of  operational  use  was  required  as  a  deliverable  for 
Milestone  A.  It  was  also  noted  that  although  specific  documents  are  required  for  transition  out  of 
the  Material  Solution  Phase  [DAU,  2010],  it  was  unclear  on  how  these  translate  into 
requirements  and  if  these  requirements  would  have  a  level  of  traceability  back  to  the  original 
inputs  drivers. 

d.  Technology  Development  Phase 

The  primary  focuses  of  the  Technology  Development  phase  are  to  prepare  the  system  and 
its  underlying  technology  ready  for  fonnal  program  initiation  at  Milestone  B.  The  key  activities 
and  work  products  of  this  process  are  shown  in  Figure  6.  This  phase  also  follows  the  V  process, 
starting  with  an  assessment  of  the  user  needs  and  culminates  with  demonstrating  or  modeling  the 
system  to  verify  compliance  with  the  specification  and  finally,  demonstrating  the  system  concept 
against  the  user  needs.  On  the  surface,  this  sounds  appropriate;  however,  two  major  issues  arise. 
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Figure  6.  Technology  Development  Phase 

This  Technology  Development  Phase  further  develops  the  system  design  and  required  technologies. 

The  first  concern  is  that  modeling  is  considered  as  a  valid  alternative  to  prototyping  with 
no  caveats  as  to  when  it  is  or  is  not  appropriate  to  use  models  as  a  substitute  for  prototypes.  The 
use  of  modeling  has  been  pushed  over  the  years  as  a  way  of  reducing  testing  costs  and  has  come 
a  long  way  in  terms  of  being  able  to  provide  useful  information.  Development  and  Contractor 
Testing  results  are  used  in  combination  with  system  models  to  predict  performance  of  the  overall 
system  without  requiring  system  level  testing.  While  this  approach  has  proven  to  be  effective  for 
predicting  operational  effectiveness  in  many  systems,  accurate  models  for  operational  suitability 
have  proven  elusive. 

One  thing  to  note  here  is  the  Weapon  Systems  Acquisition  Reform  Act  of  2009  - 
(WSARA)  which  requires  competitive  prototyping  prior  to  Milestone  B.  On  the  surface,  this 
looks  to  be  moving  in  the  right  direction.  Unfortunately,  if  the  prototypes  are  only  compared  on 
the  basis  of  the  KPPs  and  these  KPPs  do  not  encompass  the  full  range  of  requirements  including 
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suitability  and  interoperability,  then  the  full  potential  is  wasted.  No  guidance  could  be  found  that 
specified  the  full  range  of  testing  that  should  be  done  on  prototypes  prior  to  Milestone  B  and 
how  this  data  should  be  used  in  a  down-select  decision.  In  addition,  this  requirement  only 
applies  to  Major  Defense  Acquisition  Programs  (MDAPs).  MDAPs  are  defined  as  AC  AT  I 
programs  which  is  based  on  guideline  dollar  values  provided  by  the  USD  (AT&L)  and  decided 
upon  by  the  program  office.  Dollar  value  distinctions  for  ACAT  I  programs  are  total  expenditure 
for  research,  development,  test  and  evaluation  (RDT&E)  of  more  than  $365  million  in  fiscal 
(FY)  2000  constant  dollars  or,  for  procurement  of  more  than  $2. 190  billion  in  FY  2000  constant 
dollars  [DoDI  5000.02,  2008].  For  programs  with  a  smaller  expected  expenditure,  the 
Acquisition  Categories  (ACAT)  level  is  lower.  Lower  ACAT  levels  result  in  lower  Decision 
Authority  levels  required  for  acquisition  decision-making. 

e.  Engineering  and  Manufacturing  Development  Phase 

The  Engineering  and  Manufacturing  Development  phase  is  focused  on  implementation  of 
the  design  and  developing  sufficient  test  data  upon  which  to  ensure  that  the  system  is  ready  to 
proceed  to  operational  test.  The  key  activities  and  work  products  of  this  process  are  shown  in 
Figure  7.  At  this  point,  the  majority  of  the  system  level  data  should  be  based  upon  prototype 
testing  rather  than  modeling. 

A  key  issue  in  this  phase  is  the  level  of  user  involvement.  The  authors  had  some 
concerns  as  to  the  ability  of  CNO  (N8)  to  act  as  a  user  representative  with  respect  to  issues  that 
are  close  to  the  actual  operational  environment.  In  the  CBA  and  MSA  phases,  having  CNO  (N8) 
act  as  the  user  representative  was  appropriate  since  the  focus  was  on  system  level  interactions. 

In  this  phase,  user  input  is  needed  to  resolve  operational  usage,  training,  and  human  interface 
issues.  CNO  (Nl)  has  been  designated  as  the  lead  for  Human  Systems  Interfaces  (HSI)  although 
their  focus  is  on  manpower  requirements,  rather  than  human-machine  interactions.  The  only 
way  to  do  a  proper  assessment  of  these  issues  is  to  have  prototypes  of  the  system  or  key 
components  tested  by  actual  users.  The  only  fonnal  mechanism  for  this  in  the  current  process  is 
the  Operational  Assessments  performed  by  OPTEVFOR  but  no  across-the-board  requirement 
could  be  found  for  this 
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Figure  7.  Engineering  and  Manufacturing  Development  Phase 

This  Engineering  and  Manufacturing  Development  phase  is  focused  on  implementation  of  the  design  and  developing 
sufficient  test  data  upon  which  to  ensure  that  the  system  is  ready  to  proceed  to  operational  test. 

This  also  raises  the  issue  of  a  lack  of  an  independent  assessment  of  the  readiness  for 
operational  test  early  in  this  phase.  An  Operational  Test  Readiness  Review  (OTRR)  is  held 
immediately  prior  to  Milestone  C  and  the  Assessment  for  Operational  Test  Readiness  (AOTR)  is 
held  for  some  programs  after  Milestone  C  and  these  reviews  are  focused  on  assessing  if  the 
system  is  likely  to  achieve  the  operational  suitability  and  effectiveness  goals  in  OT.  Having  a 
preliminary  independent  AOTR  performed  early  in  the  Engineering  &  Manufacturing 
Development  phase  has  the  potential  to  identify  problems  in  the  design  early  enough  to  make 
changes,  or  identify  if  DT&E  data  is  insufficient  early  enough  to  plan  additional  testing. 
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f  Assessment  for  Change  Potential  and  Ease  of  Change 


In  preparation  for  the  next  phase,  the  authors  conducted  an  assessment  of  each  step  that 
was  identified  for  each  of  these  phases  for  the  relative  potential  for  improvement  and  leverage 
for  each  step.  The  results  of  this  assessment  are  shown  in  Figure  8  The  color  scheme  of  the 
graph  is  intended  to  communicate  that  steps  that  are  in  the  upper-right -hand  corner  are  the  best 
candidates  for  change  and  the  lower-left-hand  comer  indicates  poor  candidates  for  change.  This 
initial  evaluation  was  based  on  the  authors’  understanding  of  the  SE  process  and  investigation  of 
the  researched  materials. 
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•  ITR  -  Initial  Technical  Review 

•  PRR  -  Production  Readiness  Review 

•  PRE  -  Capability  Based  Assessment 

•  PRE  -  Generate  ICD 

•  MSA  -  Analyze  Component  Capabilities 

•  MSA  -  Prepare  TES 

•  MSA  -  Prepare  Draft  CDD 

•  TDS  -  Revise  Performance  Attributes 

•  EMD  -  Verify  Functionality  Compliance 

•  EMD  -  Revise  Performance  Attributes 

•  ASR  -  Alternate  Systems  Review 

•  TDS  -  Demo  System  vs  Needs 

•  SRR  -  System  Requirements  Review 

•  SFR  -  System  Functional  Review 

•  SVR  -  System  Verification  Review 

•  PDR  -  Preliminary  Design  Review 

•  CDR  -  Critical  Design  Review 

•  TRR  -  Test  Readiness  Review 

•  EMD  -  Demo  System  vs  Needs 

•  MSA  -  Analyze  Concept  Capabilities 

•  MSA  -  Verify  Concept  Performance 

•  MSA  -  Select  Preferred  Concept 

•  TDS  -  Approve  CDD 

•  EMD  -  Individual  Cl  Verification 

•  EMD  -  Approve  CPD 

•  MSA  -  Prepare  TDS 

•  TDS  -  Develop  Functional  Definitions 

•  TDS  -  Decompose  Funct.  Definitions 

•  TDS  -  Demo  System  vs  Specs 

•  TDS  -  Establish  SFB 

•  TDS  -  Establish  SAB 

•  Functional  Configuration  Audit 

•  EMD  -  Develop  Build-to  Docs 

•  EMD  -  Verify  Performance  to  Specs 

•  EMD  -  Set  Product  Support  Strategy 

•  MSA  -  Analyze  System  Needs 

•  MSA  -  Define  Supportability  Objectives 

•  MSA  -  Evaluate  Support  Capabilities 

•  TDS  -  Analyze  System  Needs 

•  TDS  -  Prepare  TEMP 

•  TDS  -  Formulate  RAM  Strategy 

•  TDS  -  Refine  Supportability  Needs 

•  TDS  -  Develop  Support  Strategy 

•  EMD  -  Develop  Support  Plan 

•  EMD  -  Demo  Support  Capability 

•  MSA  -  Decompose  Functional  Definitions 

•  TDS  -  Design  System 

•  TDS  -  Develop  Technologies 

•  TDS  -  Demo  Technologies 

•  EMD  -  Fabricate,  Assemble  to  Docs 

•  MSA  -  Develop  Concept  Objectives 

•  MSA  -  Develop  Functional  Definitions 

•  TDS  -  Develop  System  Specification 

•  TDS  -  Demo  System  vs  Plan 

•  TDS  -  Refine  System  Specification 

•  TDS  -  CDD  Joint  Staff  Review 

•  EMD  -  CPD  Joint  Staff  Review 

•  MSA  -  Develop  Concepts  &  Technologies 

Figure  8.  Assessment  of  Potential  and  Ease  of  Change 

Each  of  the  steps  identified  on  the  previous  figures  were  assessed  for  their  potential  for  improvement  and  ease  of 
change.  The  lower  left  corner  corresponds  to  steps  in  which  changes  are  likely  to  be  difficult  to  implement  due  to 
high  cost  or  higher  levels  of  approval  in  addition  to  having  a  low  potential  for  improvement.  These  steps  are 
labeled  in  red  as  being  poor  candidates  for  change.  The  upper  right  corner  is  comprised  of  steps  that  would  require 
relatively  lower  levels  of  approval  and  funding  to  change  and  have  a  high  likelihood  of  making  a  significant 
improvement  to  the  overall  process.  These  steps  are  labeled  in  green  as  being  excellent  candidates  for  change.  The 
areas  in  the  middle,  shown  in  yellow,  are  steps  that  fall  in  between  these  two  extremes. 
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Potential  for  Improvement  is  defined  as  the  importance  of  the  step  to  the  requirements 
process  in  terms  of  its  ability  to  predict  or  prevent  problems.  The  left-hand  column  of  the  chart 
shows  steps  that  do  not  appear  to  have  many  shortcomings  from  a  Systems  Engineering 
standpoint.  Also,  improvements  that  are  made  to  these  steps  will  not  have  a  substantial 
improvement  upon  the  overall  results  of  the  process,  which  is  defined  as  ensuring  that  systems 
pass  operational  test.  By  contrast,  the  right-hand  column  shows  steps  that  appear  to  not  be 
working  as  intended  such  as  the  design  reviews  that  are  supposed  to  be  ensuring  that  programs 
do  not  move  to  the  next  stage  in  the  process  if  they  are  not  ready.  This  right-hand  column  also 
indicates  that  making  improvements  to  those  steps  should  have  a  direct  improvement  on  the 
overall  process,  more  specifically,  a  higher  likelihood  of  passing  OT&E. 

Ease  of  Change  is  defined  as  our  ability  to  effect  significant  change  to  the  step  and  degree 
to  which  that  change  will  be  consistent  for  a  wide  variety  of  programs.  Steps  that  have  more 
latitude  and  are  tailored  to  the  program  are  going  to  be  more  difficult  to  improve  because 
changes  to  those  steps  will  affect  different  programs  in  different  ways.  Changes  to  a  step  may 
improve  the  results  of  some  systems  but  may  actually  hurt  other  systems.  In  addition,  steps  that 
are  perfonned  by  contractors  or  other  organizations  are  going  to  be  more  difficult  to  change 
because  we  have  less  control  over  how  those  steps  are  performed.  Additionally,  changes  that 
require  additional  funding  are  more  difficult.  Therefore,  the  bottom  row  is  those  steps  that  are 
going  to  be  the  most  difficult  or  expensive  to  change.  The  top  row  is  those  steps  that  should  be 
easier  to  change. 

g.  Summary  of  Systems  Engineering  and  Acquisition  Process  Guidance  Research 

One  finding  concluded  from  the  research  was  that  there  is  a  significant  amount  of  latitude 
built  into  the  Defense  Acquisition  System.  This  latitude  or  flexibility  was  intentional  in  the 
design  of  the  process  based  on  the  understanding  that  the  process  needed  to  be  tailored  to  the 
type  of  system,  level  of  technology  development  that  was  necessary  and  other  factors.  That 
being  said,  there  is  a  limit  on  the  amount  of  tailoring  that  is  allowed.  One  example  of  this 
limitation  is  in  the  checklists  that  are  used  for  the  design  and  technical  reviews.  In  the  past  these 
checklist  were  modifiable  and  are  now  provided  as  intentionally  locked.  This  was  done  to 
prevent  the  program  from  changing  a  wording  of  a  question  in  order  to  allow  a  satisfactory 
response  to  a  question  in  order  to  eliminate  oversight.  Questions  that  do  not  apply  to  a  program 
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can  still  be  marked  as  being  Not  Applicable  and  will  not  negatively  affect  the  “score”  for  that 
category;  however,  marking  a  question  as  Not  Applicable  still  allows  an  oversight  body  to 
question  whether  a  question  is  truly  outside  the  system  scope.  This  does  raise  the  question  of 
how  are  programs  using  this  latitude,  and  whether  programs  are  using  the  latitude  or  flexibility 
allowed  by  the  process  or  are  merely  tailoring  the  process  to  make  it  easier  for  those  programs  to 
pass  through  the  gates? 

Another  finding  showed  a  lot  of  emphasis  was  placed  on  the  Defense  Acquisition  System 
measurable  performance  parameters.  KPPs  are  just  an  example  of  a  high  level  of  these 
parameters  that  are  used  for  measuring  perfonnance  of  a  system;  there  are  many  other 
requirements  that  are  derived  from  those  KPPs.  This  is  done  to  facilitate  testing  that  can  be 
performed  at  a  lower  level  in  order  to  predict  higher-level  system  perfonnance.  However,  some 
requirements  are  very  difficult  to  quantify  and  if  system  engineers  only  look  at  the  measurable 
requirements  and  do  not  take  into  account  the  context  of  those  requirements;  these  developers 
may  believe  that  they  are  meeting  the  system  requirements,  when  in  fact  they  are  not.  This  raises 
the  question  of  whether  the  intent  of  the  requirement  is  getting  lost  as  system  developers  move 
down  through  the  decomposition  process. 

Another  issue  is  the  lack  of  communication  between  the  development  community,  user 
community  and  operational  test  communities.  There  was  little  mention  of  direct  operational  test 
community  involvement  prior  to  the  Test  Readiness  Review  even  though  the  planning  for  the 
Operational  Testing  starts  earlier  in  the  process.  The  research  indicated  that  the  development 
community  and  the  operational  test  community  essentially  work  in  parallel.  The  process  by 
which  the  operational  testers  develop  the  testing  criteria  based  upon  the  user  needs  is  very 
similar  to  how  the  system  developers  derive  the  system  requirements  and  earlier  communication 
between  these  teams  should  improve  the  accuracy  of  both  those  processes. 

2.  Operational  Test  Process  Research 
a.  Research  Methodology  and  Objectives 

As  this  project  is  in  support  of  naval  acquisition,  the  research  methodology  was  to  review 
the  existing  Defense  Acquisition  University  (DAU)  T&E  Management  Guide  and  the  existing 
OT&E  process  of  COMOPTEVFOR,  the  Navy’s  OTA.  The  objective  was  to  identify  the 
existing  process  steps  including  inputs  and  outputs,  and  to  identify  participants.  Further,  the  goal 
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was  to  critically  review  the  regulations  for  weaknesses  and  identify  potential  areas  for 
improvement. 

In  order  to  review  the  existing  OT&E  process,  the  following  documents  currently 
providing  guidance  to  naval  testers  were  reviewed: 

•  Defense  Acquisition  University  (DAU)  T&E  Management  Guide 

•  COMOPTEVFOR  Instruction  3980.1  Operational  Test  Director’s  Manual 

•  COMPOPTEVFOR  Policy  and  Infonnation  Notice  05-1A,  Mission-Based  Test  Design, 
The  Operational  Test  and  Evaluation  Framework,  and  Integrated  Test 

•  COMOPTEVFOR  Policy  and  Infonnation  Notice  05-2,  Critical  Operational  Issue  Risk 
Assessment  Reporting  Methodology 

•  COMOPTEVFOR  Policy  and  Infonnation  Notice  10-01,  Operational  Reporting 
Guidance  and  Procedure. 

The  documents  provide  information  on  the  role  of  COMOPTEVFOR,  apparent  overlap  of 
Developmental  Test  and  Evaluation  (DT&E)  with  OT&E,  OT&E  foundation  documents,  the 
Test  and  Evaluation  Master  Plan  (TEMP),  and  the  test  planning  process.  The  overall  OT&E 
planning  process  is  documented  in  Appendix  B. 

b.  Types  of  Operational  Test  and  Evaluation 

Operational  Test  and  Evaluation  (OT&E)  can  be  divided  into  two  phases:  OT&E  before 
Full  Rate  Production  (FRP)  and  OT&E  after  FRP.  OT&E  before  FRP  consists  of  Early 
Operational  Assessments  (EOA),  Operational  Assessments  (OA),  and  Initial  Operational  Test 
and  Evaluation  (IOT&E).  IOT&E  is  conducted  late  during  low  rate  production  when  actual  test 
assets  are  available  in  support  of  a  full  rate  decision  review.  After  FRP,  all  OT&E  is  referred  to 
as  Follow-on  Operational  Test  and  Evaluation  (FOT&E).  It  is  conducted  using  fielded 
production  systems  with  modifications,  upgrades  or  increments  fixing  issues  that  were  found 
during  IOT&E. 

c.  Operational  Test  and  Evaluation  Planning 

OTAs  usually  become  involved  during  a  system's  life  cycle  during  the  program’s 
evaluation  concepts.  It  is  during  this  time  that  OTAs  begin  to  develop  strategies  for  conducting 
operational  tests.  The  system  Program  Manager  (PM)  working  with  the  T&E  Working-level 
Integrated  Product  Team  (T&E  WIPT)  providing  support,  is  responsible  for  producing  the  Test 
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and  Evaluation  Master  Plan  (TEMP).  All  T&E  stakeholders  participate  early  in  the  T&E 
strategy  development  and  make  timely  updates.  Stakeholders  include  representatives  from 
Under  Secretary  of  Defense  for  Acquisition,  Technology  and  Logistics  and  Director,  Operational 
Test  and  Evaluation  (DOT&E).  For  programs  on  the  Office  of  the  Secretary  of  Defense  (OSD) 
T&E  oversight  list,  DOT&E  and  the  Director  Developmental  Test  and  Evaluation  (DDT&E) 
approve  the  TEMP  [Defense  Acquisition  Guide,  Chapter  9].  For  all  other  programs,  the 
Component  Acquisition  Executive  (CAE)  or  designated  representative  approves  the  TEMP 
[Defense  Acquisition  Guide,  Chapter  9].  OTAs  are  members  of  the  T&E  WIPT,  working 
together  they  are  responsible  for  developing  Part  III  (Test  and  Evaluation  Strategy)  and  Part  IV 
(Resource  Summary)  of  the  TEMP.  Part  III  identifies  Critical  Operational  Issues,  future  OT&E 
limitations,  and  any  Live  Fire  Test  and  Evaluation  (LFT&E).  COIs  define  operational 
effectiveness  and  operational  suitability  for  a  given  program.  Prior  to  IOT&E,  COIs  are  assessed 
by  evaluating  risks  associated  with  each  COI.  During  IOT&E,  all  COI  capabilities  and  functions 
will  be  examined.  COIs  are  resolved  as  SAT,  UNSAT,  or  unresolved.  UNSAT  or  unresolved 
COIs  will  be  re-evaluated  during  FOT&E.  Figure  9  shows  the  flow  as  to  how  COMOPTEVFOR 
plans  OT&E  for  the  Navy. 


Figure  9.  Mission  Based  Test  Design  and  Integrated  Test  (MBTD/IT)  Process 

Process  for  how  COMOPTEVFOR  plans  OT&E  for  the  Navy,  taken  from  COMOPTEVFOR  Instruction  3980.1, 

2008. 
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COMOPTEVFOR,  the  Program  Manager  (PM),  and  other  key  participants  must  complete 
and  agree  on  the  product  of  a  mission  analysis  for  the  system  [COMOPTEVFOR,  2006]. 

Mission  Based  Test  Design  (MBTD)  planning  process  entry  point  is  between  Milestone  A  and 
Milestone  B,  but  a  post  Milestone  B  entry  is  fine.  Mission  analysis  enables  direct  traceability  of 
any  system  enhancement,  risk  area,  or  deficiency  discovered  during  testing  to  a  mission  or 
missions.  During  steps  1-6  of  the  MBTD,  COMOPTEVFOR  and  the  PM  conduct  a  mission 
analysis  where  they  identify  the  mission  COIs  of  a  system.  The  mission  COIs  are  identified  by 
conducting  an  analysis  of  available  documents  such  as  the  Operational  Requirements  Document 
(ORD),  Capabilities  Documents,  and  the  Concept  of  Operations  (CONOPS)  to  identify  the 
mission  areas  for  the  system  under  test.  They  analyze  each  mission  area  to  define  tasks  required 
to  support  the  mission.  Once  the  subtasks  are  identified  they  then  establish  conditions  for  each 
subtask  and  allocate  standards  to  each  task.  The  tasks  and  subtasks  are  correlated  to  specific 
capabilities/requirements.  Steps  1-6  of  Figure  9  provide  the  Operational  Test  Director  (OTD) 
with  the  prerequisites  for  stand-alone  OT  planning  (Steps  7-11  of  Figure  9).  At  this  point  they 
develop  additional  operational  attributes  and  standards.  Test  procedures  are  then  written  for  each 
standard  or  qualitative  capability  considering  the  tasks,  subtask  and  conditions  associated  with 
that  capability.  Using  the  test  procedures,  data  requirements  are  derived,  test  scenarios  built,  and 
test  resource  requirements  are  detennined.  The  OTD  will  then  look  for  opportunities  for  DT/OT 
integrations  where  they  can  plan  to  conduct  EOAs  and  OAs.  Data  collected  during  EOA  and  OA 
is  then  analyzed  before  IOT&E.  During  IOT&E,  tests  are  conducted,  data  is  collected  and 
analyzed.  Once  the  analysis  is  complete  an  effectiveness  and  suitability  detennination  is  made. 
The  key  takeaways  of  Figure  9  are  steps  1-11.  The  mission  analysis  (Steps  1-6)  is  important 
because  this  is  where  all  of  the  tasks,  subtasks,  capabilities,  and  requirements  that  a  system  under 
test  will  have  to  meet.  Using  the  outputs  of  the  mission  analysis  provides  the  OTDs  the 
information  needed  for  them  to  plan  and  execute  OT&E  (Steps  7-1 1). 

d.  Phases  of  Operational  Test  and  Evaluation 

OT&E  can  be  conducted  throughout  the  acquisition  life  cycle  of  a  system.  Before  any 
OT  can  be  conducted,  documentation  for  major  systems  and  those  designated  y  the  Director, 
Operational  Test  and  Evaluation  (DOT&E)  for  oversight  must  be  sent  to  the  Office  of  the 
Secretary  of  Defense  (OSD)  for  approval  before  the  testing  can  be  conducted  or  the  systems  can 
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be  cleared  to  proceed  Beyond  Low  Rate  Initial  Production  (BLRIP)  [Guidebook,  2010].  Figure 
10  shows  how  OT&E  relates  to  the  Milestone  process. 


Figure  10.  OT&E  Relationship  to  the  Milestone  Process 

Process  for  how  COMOPTEVFOR  plans  OT&E  for  the  Navy,  taken  from  COMOPTEVFOR  Instruction  3980.1, 

2008. 


During  material  solution  analysis  and  technology  development  an  Early  Operational 
Assessment  (EOA)  can  be  conducted.  EOAs  are  focused  on  investigating  the  deficiencies 
identified  during  the  mission  area  analysis.  OTAs  involve  themselves  during  these  assessments 
so  that  they  can  validate  the  OT&E  requirements  for  future  testing  and  identify  issues/criteria 
that  can  only  be  resolved  through  OT&E.  This  helps  initiate  early  test  resource  planning.  An 
early  assessment  can  provide  data  to  support  a  decision  on  whether  the  technology  is  sufficiently 
mature  to  support  entry  into  the  next  developmental  phase  [DAU  Guidebook,  2010].  In  addition, 
EOA  can  be  difficult  since  testing  is  done  on  mock-ups  or  using  models/simulations.  Documents 
are  also  reviewed  and  both  contractor/ government  data  are  evaluated.  This  leads  to  inputs  being 
provided  into  the  Test  and  Evaluation  Master  Plan  (TEMP).  Special  logistics  problems,  program 
objectives,  program  plans,  performance  parameters,  and  acquisition  strategy  are  areas  of  primary 
influence  to  the  operational  tester  during  this  phase  and  must  be  carefully  evaluated  to  project  the 
system’s  operational  effectiveness  and  suitability  [DAU  Guidebook,  2010]. 
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Additional  EOAs  or  Operational  Assessments  (OA)  may  be  conducted  during  the 
Engineering  and  Manufacturing  Development  phase.  OAs  are  conducted  to  facilitate 
identification  of  the  best  design,  indicate  the  risk  level  of  perfonnance  for  this  phase  of  the 
development,  examine  operational  aspects  of  the  system’s  development,  and  estimate  potential 
operational  effectiveness  and  suitability  [DAU  Guidebook,  2010].  The  data  collected  during 
these  OAs  can  be  reviewed  and  used  to  detennine  whether  or  not  a  system  is  ready  to  move  into 
the  development  of  the  Engineering  Development  Model  (EDM).  EOAs  and  OAs  are  conducted 
with  DT&E  testers,  contractors  and  OT&E  testers 

During  the  Production  and  Deployment  phase  (P&D),  equipment  that  has  been  formally 
certified  by  the  Program  Manager  as  being  ready  for  OT&E  is  tested  [DAU  Guidebook,  2010], 
Initial  Operational  Test  and  Evaluation  (IOT&E)  is  then  conducted  by  user  organizations  in  a 
field  exercise  to  examine  the  organization  and  doctrine,  integrated  logistic  support,  threat, 
communications,  command  and  control,  and  tactics  associated  with  the  operational  employment 
of  the  system.  After  the  Full  Rate  Production  (FRP)  decision,  Follow-on  Operational  Test  and 
Evaluation  (FOT&E)  is  conducted  to  refine  effectiveness  and  suitability  estimates,  assess 
performance  not  evaluated  during  IOT&E,  evaluate  new  tactics  and  doctrine,  and  assess  the 
impacts  of  system  modifications  or  upgrades  of  issues  found  during  earlier  EOAs,  OAs  and 
IOT&E.  FOT&E  is  conducted  by  the  users  in  a  operational  environment.  Overall  the  OTA  is 
required  to  provide  reports  of  OT&E  results  in  support  of  Milestone  B  and  C,  in  addition  to  the 
IOT&E  report  required  for  the  Full  Rate  Production  Decision  Review  (FRPDR)  [DAU 
Guidebook,  2010]. 

e.  Summary  of  Operational  Test  Planning  Process  Research 

COMOPTEVFOR  is  an  independent  agent  for  OT&E  in  the  Navy’s  acquisition  process 
whose  mission  is  to  “independently  and  objectively  evaluate  the  operational  effectiveness  and 
suitability  of  new  and  improved  war  fighting  capabilities’’  [COMOTEVFORINST  3980.1,  2008: 
1-1].  OT&E  is  to  be  conducted  in  as  near  a  real  operational  environment  as  possible  with  fleet 
personnel  operating  and  maintaining  the  system  under  test.  As  systems  are  passing 
Developmental  Test  and  Evaluation  (DT&E)  yet  failing  OT&E,  it  is  important  to  understand  the 
key  concepts  pertaining  to  developmental  testing.  DT&E  is  planned  and  conducted  by  the 
Design  Agent  (DA)  responsible  for  the  development  of  the  system  and  Program  Executive 
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Office  (PEO)  in  case  of  the  Navy  who  assigns  a  Government  agency  to  conduct  DT&E,  such  as 
the  Naval  Surface  Warfare  Centers  (NSWC),  Naval  Air  Warfare  Centers  (NAWC)  and  Naval 
Underwater  Warfare  Centers  (NUWC).  These  tests  can  be  conducted  at  a  contractor’s  facility,  a 
government  facility  or  out  in  the  field.  One  finding  showed  that  the  viewpoints  between  DT&E 
and  OT&E  are  different.  They  both  differ  in  the  way  tests  are  conducted,  in  what  is  being  tested, 
and  in  the  evaluation  criteria.  Table  2  reflects  the  key  differences. 


Table  2.  DT&E  and  OT&E  Differences 

This  table,  taken  from  COMOPTEVFORINST  3980.1,  summarizes  the  differences  between  Developmental  Test  and 
_ Operational  Test. _ 


How  Tests  are  Conducted 

DT&E  testing  is  properly  conducted: 

•  In  a  controlled  environment  that  minimizes  the 
chance  that  unknown  or  unmeasured  variables 
will  affect  system  performance 

•  By  technical  personnel  skilled  at  “tweaking”  to 
maximize  performance 

•  Against  simulated  threats  tailored  to 
demonstrate  various  aspects  of  specified  system 
technical  performance 

OT&E  testing  is  properly  conducted: 

•  In  an  operationally  realistic  environment  (e.g.,  high 
seas,  temperature  extremes,  high  density 
electromagnetic  environments)  under  conditions 
simulating  combat  stress  and  peacetime  conditions 

•  With  Fleet  operators  and  maintenance  personnel 

•  Against  threats  which  replicate,  as  closely  as  possible, 
the  spectrum  of  operational  characteristics 

•  Using  Fleet  tactics 

Testing  Subject/Topic 

DT&E  tests  a  weapon  or  a  “blackbox,”  whatever  the 
development  program  involves.  (Seldom  does  a 
development  program  involve  a  complete  weapon 
system) 

OT&E  tests  total  weapon  systems.  If  a  missile  is  being 
developed,  OT&E  does  not  test  only  the  missile  itself,  but 
rather  the  missile  system,  which  includes  the  firing 
platform;  that  platform’s  detection,  classification,  and 
targeting  systems;  the  people  who  man  it;  logistical  support; 
interfacing  equipment;  etc. 

Evaluation  Criteria 

DT&E  -  Technical  criteria  are  parameters  measured 
during  controlled  DT&E  tests. 

OT&E  -  Operational  criteria  are  the  CNO-provided 
minimum  acceptable  operational  performance  requirements 
(older  programs)  or  measures  of  effectiveness/suitability 
(newer  programs),  or  thresholds,  which  quantify  the  Critical 
Operational  Issues  (COI). 

Measurement  and  Frequency 

DT&E 

•  The  DA  generally  knows  what  he/she  wants  to 
measure  (some  particular  parameter:  launch 
velocity;  the  number  of  g’s  pulled  as  the  missile 
acquires;  time  to  climb;  etc.). 

•  DT&E  tests  are  structured  to  hold  many  things 
constant,  isolate  others,  and  allow  measurement 
of  one  or  two  parameters  of  interest. 

•  It  is  generally  possible  to  verify  data 
statistically  through  replication  of  tests. 

OT&E 

•  It  is  often  not  possible  to  specify  measurements. 

•  The  objective  is  often  simply  to  create  combat 
conditions  as  closely  as  possible  and  record  data  as 
events  unfold. 

•  For  aviation  OT&E,  with  highly  time-compressed  test 
events  and  a  high  cost  for  OT&E,  it  is  mandatory  that 
OTDs  know  exactly  what  parameters  of  their  system 
must  be  examined  to  resolve  the  specified  COI. 

•  OT&E  cannot  enjoy  the  luxury  of  isolating  a  variable. 
Methods  of  data  capture  must  be  devised  during 
operational  evolutions  or  during  post-operation 
analysis. 

•  It  is  often  not  possible  to  replicate  data  because 
interactions  during  tests  are  unique. 
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One  recommendation  to  improve  the  test  planning  process  would  be  for  OT&E  personnel 
to  work  with  the  DT&E  personnel  so  that  DT&E  can  be  conducted  in  more  of  an  operational  test 
setting  and  allow  for  the  planning  of  test  phases  that  provide  the  necessary  data  to  satisfy 
common  needs  of  the  DA  and  the  OT&E  agency.  This  would  have  the  effect  of  reducing  the 
differences  between  DT&E  and  OT&E  cited  above,  and  possibly  allow  system  deficiencies  to  be 
discovered  earlier  in  the  process  during  DT&E. 

Another  possible  improvement  in  the  test  planning  process  is  in  the  tracking  and  review 
of  Critical  Operational  Issues  (COIs).  COMOPTEVFOR  develops  the  COIs  for  each  program 
and  publishes  them  in  the  Test  and  Evaluation  Master  Plan  (TEMP).  The  COIs  are  linked  to 
CNO  requirements  established  in  the  Operational  Requirements  Document  (ORD),  Analysis  of 
Alternatives  (AoA),  Initial  Capabilities  Document  (ICD),  Capability  Development  Document 
(CDD),  and  Capability  Production  Document  (CPD).  COIs  define  the  operational  effectiveness 
and  operational  suitability  for  a  given  system.  They  are  assessed  and  will  be  resolved  as  SAT, 
UNSAT  or  unresolved.  COIs  need  to  be  defined  correctly,  otherwise  incorrectly  defined  COIs 
may  remain  throughout  the  test  planning  and  test  execution.  Another  recommendation  would  be 
for  a  fonnal  role  in  the  Systems  Engineering  Technical  Review  (SETR)  for  the  OT  community  in 
which  COIs  are  indentified  and  validated  with  the  users  and  traced  to  the  system  requirements  at 
several  reviews  throughout  the  development  phase. 

3.  Operational  Test  Failure  Research 

a.  Research  Methodology  and  Objectives 

There  were  two  principal  objectives  to  this  task.  First,  research  was  conducted  for 
historical  infonnation  that  could  give  some  insight  as  to  why  some  programs  fail  operational 
testing.  This  was  accomplished  via  a  search  to  find  publically  available  literature  applicable  to 
operational  testing  and  examining  them  to  identify  possible  issues  or  challenges  programs  in 
general  have  experienced.  The  second  objective  in  this  task  was  to  identify  actual  programs  that 
had  recently  experienced  problems  during  operational  test  and  examine  them  to  detennine 
potential  sources  of  failure.  This  was  accomplished  by  requesting  documentation  from  programs 
via  contacts  held  by  the  authors  and  contacts  provided  by  the  project  sponsor,  ASN  (RDA) 
CHSENG.  The  overall  goal  with  this  combined  approach  was  to  locate  and  study  previously 
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performed  research  on  this  subject  (or  similar  subject)  to  identify  potential  root  causes,  and  then 
compare  them  to  specific  programmatic  documentation  to  find  any  similarities  or  differences. 

To  accomplish  the  first  objective  of  this  task,  an  extensive  search  of  publicly  available 
papers,  articles,  and  reports  was  complete.  These  included  but  were  not  limited  to: 

•  Government  Accountability  Office  (GAO)  reports 

•  Senate  Anned  Committee  reports. 

•  Defense  Science  Board  reports 

•  Various  topic  papers  from  the  NPS  Library 

Most  of  the  literature  found  was  very  general  and  not  program  specific,  but  it  provided 
good  insight  to  the  historical  perfonnance  of  programs  undergoing  operational  testing.  From  this 
research  several  conclusions  could  be  made  regarding  the  sources  of  OT  failures.  These  findings 
and  conclusions  will  be  discussed  in  the  following  sections. 

b.  Operational  Suitability 

First,  while  several  programs  were  shown  to  have  failed  OT  due  to  a  lack  of  operational 
effectiveness  (failure  to  meet  performance  requirements,  functionality,  etc.),  multiple  sources 
identified  a  lack  of  operational  suitability  as  a  major  cause  of  failures  that  is  not  being  adequately 
addressed  by  programs.  Operational  suitability  includes  requirements  such  as  reliability, 
availability,  maintainability,  and  training.  These  operational  suitability  failures  are  clearly 
illustrated  in  the  “Report  of  the  Defense  Science  Board  Task  Force  on  Developmental  Test 
Evaluation”  [Defense  Science  Board  Task  Force,  2008].  This  paper  gave  a  good  overview  of  the 
historical  perfonnance  of  various  programs  during  OT  from  all  three  services  from  2001  to  2006. 
Figures  1 1,  12  and  13  are  copied  directly  from  Figures  1-3  of  the  DSBTF  Report  dated  May 
2008. 
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Program 

Service 

ACAT 

IOT&E  Result 

Reason 

FY  2001 

F-15TEWS 

USAF 

II 

Effective 

Reliability,  Maintainability,  Availability 

V-22  Osprey 

Navy 

ID 

Effective 

Reliability,  Availability,  Maintainability 
(RAM),  Human  Factors,  BIT 

Joint  Direct  Attack 

Munitions  (JDAM) 

USAF 

1C 

Effective  only  with 
legacy  fuses 

Integration  with  delivery  platforms 

M2A3  Bradley  Fighting 
Vehicle 

Army 

ID 

Effective 

Suitable 

FY2002 

Joint  Primary  Aircraft 
Training  System  (JPATS) 

USAF 

1C 

Effective  with 
deficiencies 

RAM,  Safety,  Human  Factors 

Cooperative  Engagement 
Capability  (CEC) 

Navy 

ID 

Effective 

Suitable 

Multiple  Rocket  Launcher 
System  (MLRS) 

Army 

1C 

Effective 

Suitable 

MH-60S 

Navy 

1C 

Effective 

RAM,  excessive  administrative  and  logistic 
repair  time  impacted  RAM 

FY  2003 

B-1B  Block  E  Mission 
Upgrade  Program 

USAF 

ID 

Effective 

■  16%  decrease  in  weapons  release  rate. 

1  reduction  in  accuracy  of  Mark  82  low  drag 

M  weapons.  14%  hit  rate  on  moving  targets 

Sea  wolf  Nuclear  Attack 

Submarine 

Navy 

ID 

Effective 

Suitable 

Several  requirement  thresholds  were  not 
met  but  overall  system  effective  andsuitabls 

Figure  11.  DoD  IOT&E  Results  FY  2001-2003 

This  figure,  from  Defense  Science  Board  Task  Force,  2008,  shows  the  Initial  Operational  Test  &  Evaluation  results 

from  FY2001  to  FY2003. 
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Program 

Service 

ACAT 

IOT&E  Result 

Reason 

FY  2004 

Evolved  Sea  sparrow  Missile 

Navy 

II 

Effectiveness 

unresolved 

Suitable 

Testing  was  not  adequate  to 
determine  effectiveness. 

Stryker 

Army 

ID 

Effective 

Suitable 

Advanced  SEAL  Delivery  System 
(ASDS) 

Navy 

ID 

Effective  with 

restrictions 

Effective  for  short  duration 
missions;  not  effective  for  all 
missions  and  profiles. 

Not  suitable  due  to  RAM. 

Tactical  Tomahawk 

Navy 

1C 

Effective 

Suitable 

Stryker  Mortar  Carrier-B  (MC-B) 

Army 

ID 

Effective 

RAM  and  safety  concerns. 

FY  2005 


CH-47F  Block  1 

Army 

1C 

Effective 

RAM;  communications  system  less 
suitable  than  CH-47D;  did  not 
meet  Information  Exchange 
Requirements  for  Block  1. 

F/A-22 

USAF 

ID 

Effective 

RAM;  needed  more  maintenance 
resourcesand  spare  parts;  BIT 

Joint  Stand-Off  Weapon-C 

Navy 

1C 

Not  effective  against  moderately 
hardened  targets;  mission 
planning  time  was  excessive. 

Guidcd-MLRS 

Army 

1C 

Effective 

Suitable 

High  Mobility  Attack  Rocket  System 
(HMARS) 

Army 

1C 

Effective 

Suitable 

V-22  Osprey 

Navy 

ID 

Effective 

Suitable 

EA-6B  (ICAP  III) 

Navy 

II 

Effective 

Suitable 

Figure  12.  DoD  IOT&E  Results  FY  2004-2005 

This  figure,  from  Defense  Science  Board  Task  Force,  2008,  shows  the  Initial  Operational  Test  &  Evaluation  results 

from  FY2004  to  FY2005. 
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Program 


Service 


IOT&E  Result 


Reason 


ACAT 


Cy2006 


Common  Missile  Warning 
System  (CMWS) 


Army 


1C 


Deployable  Joint  Command  and 
Control  (DX2) 


Navy 


1AM 


Integrated  Defensive  Electronic 


Navy  II 


Countermeasures 


Surface  Electronic  Warfare 
Improvement  Program  (SEW1P) 
Block  1A 


Navy 


C-130J 


USAF 


1C 


Small  Diameter  Bomb  (SOB) 
Increment  1 


USAF 


ID 


Effective  single 
ship,  Not  effective 
in  formation 


Suitable  with 
shortfalls 


Effective  with 
limitations 


Suitable  with 
limitations 


Effective  and  suitable  in  the  OIF/OEF  environment 
but  needs  further  testing  outside  of  the  OIF/OEF 
environment 


Operational  Test  Agency,  COTF,  reported  effective, 
not  suitable  BLRIP  not  complete 


Test  suspended  due  to  reliability  problems 


Block  1 A  Upgrade  does  not  make  the  AN/SLQ-32 
EWS  operationally  effective  and  suitable  but  does 
enhance  ability  to  protect  ships 


Effective  single  ship,  not  effective  in  formation  air  land 
I  air  drop,  not  effective  in  non  permissive  threat 
environment  Shortfalls  in  suitability  due  to 
maintainability  issues 


Limited  effectiveness  and  suitability  due  to  bomb  rack 
reliability  and  deficiencies  in  software  used  to  predict 
optimum  fuzing  solutions  Oct  2006  flight  operations 
suspended 


Figure  13.  DoD  IOT&E  Results  FY  2006-2008 


This  figure,  from  Defense  Science  Board  Task  Force,  2008,  shows  the  Initial  Operational  Test  &  Evaluation  results 

from  FY2006  to  FY2008. 


It  can  be  seen  that  the  majority  of  the  failures  in  OT  between  2001  and  2006  were  due  to 
a  lack  of  operational  suitability.  These  were  mostly  identified  as  reliability  shortfalls, 
maintainability  issues,  or  interoperability  problems.  The  lack  of  operational  suitability  was  also 
identified  as  a  major  contributor  to  OT  failures  by  Dr.  Charles  McQueary,  the  Director  for 
Operational  Test  and  Evaluation,  in  a  2007  interview: 
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Test  results  since  2001  show  that  almost  50  percent  of  DoD's  programs  in 

oversight  are  unsuitable  at  the  time  of  initial  operational  test  and  evaluation— IOT 

&  E— because  they  do  not  achieve  reliability  goals  [McQueary,  2008:  4]. 

The  research  suggested  several  underlying  causes  for  the  operational  suitability  failures. 
The  first  is  the  inadequate  early  investment  of  programs  into  a  robust  operational  suitability 
program.  This  could  result  in  OT  failures  due  to  inadequate,  unrealistic,  or  even  omitted 
operational  suitability  tests  or  simulations  performed  early  in  the  program  that  could  potentially 
identify  shortcomings  prior  to  comprehensive  operational  testing  [Defense  Science  Board  Task 
Force,  2008;  McQueary,  2008].  In  particular,  without  a  robust  operational  suitability  program, 
modeling  and  simulation  techniques  often  used  in  lieu  of  early  testing  may  not  adequately 
measure  operational  suitability  requirements.  Operational  suitability  can  be  a  particularly 
difficult  area  to  model  and  simulate,  and  physical  testing  can  prove  to  have  unforeseen  effects  on 
systems  not  shown  during  simulations. 

Secondly,  some  research  suggested  that  deficiencies  in  training  methods  for  both  the 
operational  use  and  the  maintenance  of  the  systems  may  be  masked  during  early  developmental 
testing.  Contractors  are  often  present  during  early  testing  and  are  readily  available  to  fix  any 
problems  that  come  up  in  early  tests  due  to  inadequate  user  training.  During  OT,  contractors  are 
usually  not  available  to  solve  problems,  and  training  deficiencies  become  much  more  apparent 
[Schrobo,  2010]. 

Finally,  many  systems  are  entering  OT  with  operational  suitability  deficiencies  that  point 
to  inadequacies  in  the  Systems  Engineering  Technical  Review  (SETR)  process.  All  programs 
must  demonstrate  they  meet  certain  stated  operational  suitability  requirements  in  order  proceed 
through  the  project  phases,  yet  in  some  cases  the  SETR  process  is  either  not  identifying 
deficiencies  in  the  system  or  is  allowing  the  system  to  proceed  to  OT  regardless  [Defense 
Science  Board  Task  Force,  2008]. 

c.  Interoperability  and  Integration 

Interoperability  and  integration  was  another  major  source  of  OT  failures  identified  in  the 
research  [GAO,  2008;  DeLaurentis,  2009].  Several  potential  sources  of  failure  were  identified, 
many  of  which  are  similar  to  the  issues  operational  suitability  suffered. 

First,  it  was  recognized  that  incorrect  or  inadequate  requirements  may  be  identified 
during  the  program’s  requirements  process.  This  may  be  due  to  the  difficulty  of  working  with 
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large  System  of  Systems  (SoS)  that  have  many  different  Technology  Readiness  Levels  (TRLs) 
amongst  its  components,  especially  if  a  legacy  system  is  incorporated.  TRLs  are  not  adequate  as 
a  metric  to  measure  system  integration,  and  may  mislead  the  program  in  assuming  that  any 
interoperability  challenges  are  minor  and  easily  handled  [Sauser  et  al,  2009].  Incorrect  or 
inadequate  requirements  may  also  be  due  to  the  lack  of  control  the  SoS  engineer  may  have  over 
the  component  systems;  while  incorrect  or  incomplete  infonnation  about  the  component  systems 
may  be  used  to  fonnulate  unrealistic  interoperability  requirements  [DeLaurentis,  2009]. 

Second,  it  is  recognized  that  there  are  several  root  causes  identified  that  were  associated 
with  interoperability,  in  particular  with  regards  to  early  testing  and/or  simulation.  Similar  to 
what  can  occur  with  operational  suitability,  early  interoperability  testing  and  simulation  could  be 
either  inadequate  or  unrealistic,  or  not  being  performed  at  all.  Unfortunately  in  large  SoSs,  early 
demonstrations  of  interoperability  are  usually  limited  to  simulations,  which  are  inadequate  and/or 
unrealistic  for  interoperability  tests.  This  makes  early  program  testing  unfeasible.  In  these 
cases,  unless  the  early  simulations  were  accurate  in  modeling  the  SoS,  deficiencies  in 
interoperability  will  likely  not  be  discovered  until  comprehensive  operational  testing. 

Finally,  the  fact  that  many  systems  are  entering  OT  with  interoperability  deficiencies, 
points  to  inadequacies  in  the  Systems  Engineering  Technical  Review  (SETR)  process. 

Assuming  the  interoperability  requirements  for  the  program  are  correct  and  realistic,  the  SETR 
process  is  either  not  identifying  deficiencies  in  the  system  or  is  allowing  the  system  to  proceed  to 
OT  regardless. 

d.  Testing  Methodologies 

While  reviewing  general  testing  methodologies  frequently  identified  within  the 
researched  literature,  an  issue  was  discovered  on  how  to  properly  implement  early  “operationally 
realistic”  testing.  Performing  early  testing  in  areas  such  as  effectiveness,  interoperability  and 
suitability  is  cited  many  times  as  a  method  of  alleviating  failures  in  OT;  however,  it  has  been 
stressed  that  such  early  testing  must  be  as  “operationally  realistic”  as  possible  in  order  to 
properly  identify  deficiencies  in  the  system.  As  discussed  earlier,  inadequate  or  unrealistic  tests 
or  simulations  performed  early  in  the  program  will  not  prevent  failures  from  occurring  during 
OT.  Unfortunately,  it  is  often  the  case  that  inadequacies  in  the  early  testing  are  not  discovered 
until  such  OT  failures  occur. 
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One  methodology  that  needs  to  be  stressed  is  the  involvement  of  OT  personnel  early  in 
the  program  [McQueary,  2008].  Early  involvement  of  OT  personnel  can  identify  requirements 
that  are  unrealistic  or  impossible  to  test  for.  By  engaging  test  personnel  early  and  often,  a 
program  can  ensure  that  the  early  testing  and  simulations  they  perform  are  as  adequate  and 
realistic  as  possible,  and  catch  any  system  deficiencies  before  they  enter  OT. 

e.  Program  Research 

To  accomplish  the  second  objective  in  this  task,  the  authors  identified  actual  programs 
that  had  recently  experienced  problems  during  operational  test  and  obtained  operational  test 
documentation  from  them  to  review.  Several  of  the  authors  had  contacts  with  applicable 
programs;  others  were  provided  by  the  project  sponsor,  ASN  (RDA)  CHSENG.  The 
documentation  received  from  all  sources  was  reviewed  and  evaluated.  Note  that  much  of  the 
program  documentation  that  was  received  was  classified;  therefore,  no  specific  (technical)  causes 
of  OT  failures  will  be  discussed  in  this  report.  Appendix  B  lists  references  for  all 
COMOPTEVFOR  reports  used  during  this  research. 

Infonnation  was  provided  for  a  total  of  6  separate  programs.  All  of  the  programs 
reviewed  encountered  major  problems  during  OT;  the  general  causes  of  their  failures  are 
summarized  in  Table  3. 

Table  3.  Summary  of  Reviewed  Programs 

This  table  summarizes  the  systems  that  were  studied  and  the  types  of  failures  that  were  identified. 
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The  review  of  the  OT  reports  and  other  provided  documentation  identified  similar  trends 
and  issues  found  during  the  literature  research.  Interoperability  and  reliability  were  cited  as  the 
most  frequent  deficiencies,  with  training  deficiencies  not  far  behind.  The  IFF,  SSDS  and  CEC 
programs  both  cited  major  problems  in  interoperability  during  OT  [COMOPTEVFOR  (MK 
XIIA),  2009;  COMOPTEVFOR  (SSDS),  2008;  COMOPTEVFOR  (CEC),  2001],  while  the  IFF, 
RMS,  and  CEC  programs  found  major  deficiencies  in  reliability,  both  with  regards  to  hardware 
and  software  [COMOPTEVFOR  (MK  XIIA),  2009;  COMOPTEVFOR  (RMS),  2007; 
COMOPTEVFOR  (CEC),  2001]  . 

The  program  and  test  community  documentation  research  matched  the  results  of  the  open 
literature  research  very  well.  While  some  programs  did  fail  OT  due  to  perfonnance  or 
functionality  deficiencies,  the  majority  of  programs  reviewed  had  greater  problems  with 
interoperability  and  operational  suitability  (reliability  and  training  in  particular). 

f  Summary  of  Open  Literature  and  Program  Documentation  Research 

Based  on  the  results  of  the  open  literature  and  program  documentation  research,  several 
general  conclusions  could  be  made  regarding  the  causes  of  OT  failures.  First,  while  many 
programs  fail  OT  due  to  operational  effectiveness  deficiencies,  the  bulk  of  the  systems  cited  in 
the  literature  and  the  specific  programs  that  were  reviewed  have  seen  failures  in  OT  due  to  a  lack 
of  operational  suitability  or  interoperability.  These  are  mission  critical  areas  of  concern; 
however,  when  compared  to  standard  perfonnance  goals  these  areas  can  be  nebulous,  difficult  to 
test  or  simulate  in  a  realistic  way,  and  may  be  prone  to  being  disregarded  due  to  programmatic 
pressures. 

Secondly,  for  all  areas  (effectiveness,  suitability,  and  interoperability),  the  inclusion  of 
OT  personnel  early  in  the  process  is  important  to  ensure  adequate,  realistic  testing  or  simulation 
is  performed  prior  to  OT  to  discover  any  deficiencies.  Any  simulations  that  are  perfonned  in 
lieu  of  early  testing  must  also  be  validated  to  ensure  deficiencies  are  being  properly  identified. 
And  finally,  there  may  be  issues  with  the  SETR  process.  In  some  cases,  deficiencies  found 
during  early  testing  or  simulations  have  not  been  identified  in  the  SETR  process.  Or  if  they 
have,  programs  somehow  are  proceeding  to  OT  regardless. 
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C.  PROJECT  RESCOPE 

1.  Bounded  Problem  Statement 

After  considering  the  advice  provided  by  the  panel  to  the  authors  at  the  first  IPR,  it  was 
decided  to  reduce  the  scope  of  the  project  by  only  considering  OT  failures  that  occurred  in  a 
single  mission  critical  area.  The  open  literature  and  program/test  documentation  research 
performed  in  the  Process  Review  and  Problem  Identification  Phase  identified  two  major  mission 
critical  areas  where  OT  failures  were  likely  to  occur:  operational  suitability  and  interoperability. 
Enough  literature  and  example  programs  had  been  identified  to  support  a  focus  in  either  area; 
however  it  was  noted  that  improving  the  operational  suitability  of  systems  had  already  been  the 
focus  of  several  government  studies  to  date  that  had  provided  their  own  conclusions  and 
recommendations.  In  an  effort  to  avoid  repeating  previous  works,  OT  failures  due  to  a  lack  of 
interoperability  were  chosen  to  be  the  focus  for  the  remainder  of  the  project. 

The  definition  of  “interoperability”  was  very  important  to  gain  this  focus  on  the  project. 
It  was  detennined  by  the  authors  that  a  comprehensive  definition  of  interoperability  would  need 
to  be  agreed  upon  for  use  during  the  remainder  of  the  efforts.  Research  into  the  available 
literature  revealed  a  wide  variety  of  interoperability  definitions  [Camey,  2005;  Morris,  2004; 
Brownsword,  2004;  Ford,  2010],  Furthermore,  it  was  discovered  that  the  definition  of 
interoperability  often  overlapped  definitions  of  the  related  concept  of  “integration,”  creating 
confusion  between  the  meanings  of  the  two  terms.  Clarification  of  the  tenns  was  also  requested 
from  the  project  sponsor.  The  sponsor  suggested  the  definitions  of:  “Interoperability  is  putting 
two  or  more  systems  together  in  any  way,  seamless  or  not  and  that  connection  allows 
perfonnance  of  a  new  requirement  for  the  systems  or  an  enhanced  requirement  in  one  of  the 
existing  systems.  Integration  is  simply  putting  two  or  more  components  together  in  a  seamless 
way  so  the  system  operation  meets  required  performance”  [McKinlay,  2010], 

Taking  into  consideration  the  researched  definitions  [Carney,  2005;  Morris,  2004; 
Brownsword,  2004;  Ford,  2010],  the  recommendation  from  the  sponsor  [McKinlay,  2010]  and 
the  definition  called  out  in  the  Department  of  Defense  Joint  Publication  [DoD  JP  1-02,  2008];  it 
was  detennined  that  the  definition  in  the  Department  of  Defense  Dictionary  of  Military  and 
Associated  Terms  [DoD  JP  1-02,  2008]  would  be  the  best  definitions  to  use  for  this  project.  The 
authors  wanted  to  comply  with  the  standard  military  definitions  such  that  this  meaning  would  be 
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common  across  military  applications  and  understandable  to  engineers  and  analysts  working  on 
military  systems.  These  definitions  are  sufficient  for  the  purpose  of  this  project  because  they 
provide  for  a  common  understanding  for  the  elements  that  make  up  a  System-of-System.  The 
definitions  provided  in  JP  1-02  and  used  in  this  project  are: 

Interoperability 

The  condition  achieved  among  communications-electronics  systems  or  items  of 
communications-electronics  equipment  when  information  or  services  can  be  exchanged  directly 
and  satisfactorily  between  them  and/or  their  users. 

Integration 

The  arrangement  of  military  forces  and  their  actions  to  create  a  force  that  operates  by 
engaging  as  a  whole. 

The  expected  deliverables  of  this  project  remained  the  same:  recommended  changes  to 
the  Systems  Engineering  (SE)  processes  in  the  Defense  Acquisition  System  in  order  to  increase 
the  percentage  of  programs  that  successfully  pass  OT&E  and  therefore  decrease  the  cost  and 
schedule  delays  associated  with  programs  failing  OT&E.  However,  with  the  decreased  scope  to 
focus  on  only  one  mission  critical  area,  the  recommended  changes  would  now  only  focus  on 
improving  interoperability  performance  during  OT&E. 

2.  Requirements  of  the  Bounded  Problem 

Once  the  scope  of  the  project  had  been  narrowed,  it  was  necessary  to  revisit  the  initial 
requirements  to  review  their  applicability  and  add  additional  requirements  that  were  discovered 
during  the  first  phase  of  the  project.  The  following  requirements  were  indentified  early  in  the 
Review  and  Problem  Identification  Phase  and  were  described  in  detail  in  Section  A  above: 

A.  Develop  a  Method  to  Track  Mission  Critical  Requirements 

B.  Explore  a  Mission  Thread  with  Respect  to  Mission  Critical  Requirements 

C.  Develop  a  Conceptual  Approach  to  Improving  OT  Results 

D.  Improve  OT  Success  Rate 

E.  Reduce  Rework  Costs 

These  requirements  still  apply  after  reducing  the  scope  of  the  project  to  focus  on  the 
interoperability  mission  critical  area.  Requirements  C,  D  and  E  could  remain  as  written, 
however  the  first  two  requirements  were  refined  as: 
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A.  Develop  a  Method  to  Track  Interoperability  Requirements 

B.  Explore  a  Mission  Thread  with  Respect  to  Interoperability  Requirements 

Additionally,  a  number  of  deficiencies  were  identified  by  the  authors  as  described  in  the 

research  section  and  these  deficiencies  must  be  taken  into  consideration  in  the  design  of  any 
solution.  These  requirements  were  added  to  form  an  expanded  requirements  list  of  the  bounded 
problem: 

A.  Develop  a  Method  to  Track  Interoperability  Requirements 

B.  Explore  a  Mission  Thread  with  Respect  to  Interoperability  Requirements 

C.  Develop  a  Conceptual  Approach  to  Improving  OT  Results 

D.  Improve  OT  Success  Rate 

E.  Reduce  Rework  Costs 

F.  Improve  Visibility  of  System  Performance  Throughout  the  Process  and  Earlier 
Detection  of  Problems 

G.  Improve  Alignment  of  Developmental  Testing  to  Operational  Testing 

H.  Improve  Alignment  of  OT&E  Critical  Operational  Issues  and  System  Design 
Requirements 

I.  Minimize  Time  and  Cost  Required  to  Implement  the  New  Process 

3.  Value  System  Design 

The  value  of  a  new  process  for  the  various  stakeholders  is  a  function  of  the  ability  of  the 
new  process  to  minimize  cost,  improve  schedule,  and  improve  the  perfonnance  of  the  system  in 
addition  to  meeting  all  required  regulations.  Additionally,  the  ease  and  speed  of  implementing 
the  changes  will  increase  the  customer  value  by  allowing  the  benefits  of  the  new  process  to  be 
achieved  earlier.  There  are  several  components  of  each  of  these  five  primary  factors  as  shown  in 
Figure  14. 
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Figure  14.  Value  System 

This  figure  shows  the  hierarchy  of  values  for  this  process  based  upon  a  study  of  the  stakeholder  needs. 


Each  stakeholder  would  assign  different  preference  weightings  on  the  relative  importance 
of  each  of  these  factors  based  upon  their  point  of  view  and  underlying  role  in  the  process.  For 
example,  the  war-fighter  would  likely  put  maximum  importance  upon  the  performance  of  the 
system  and  would  also  place  significant  importance  upon  schedule  perfonnance  since  delays  in 
system  delivery  will  deny  the  war-fighter  the  benefits  of  the  new  system.  Alternatively,  the 
program  office  is  primarily  focused  upon  meeting  the  schedule  and  cost  requirements  in  addition 
to  meeting  all  regulatory  guidance.  That  is  not  to  say  that  the  program  office  places  no 
importance  on  system  performance;  however,  they  must  balance  the  needs  of  the  war  fighter 
with  the  fixed  constraints  of  budget  and  regulations. 


43 


THIS  PAGE  INTENTIONALLY  LEFT  BLANK 


44 


III.  ANALYSIS  OF  ALTERNATIVES 


Following  the  completion  of  the  first  phase  of  the  project  and  the  subsequent  re-scope  to 
focus  on  OT  failures  due  to  interoperability,  the  down  selection  or  Analysis  of  Alternatives 
(AoA)  phase  was  begun.  The  term  AoA  is  used  throughout  the  remainder  of  this  report  and  is 
synonymous  with  the  term  down  selection.  Several  tasks  were  completed  during  this  phase. 
Alternative  solutions  were  developed  by  utilizing  a  multi-step  brainstonning  process. 

Alternative  solutions  were  compared  and  the  one  with  the  greatest  potential  to  be  successfully 
developed  and  integrated  into  the  existing  SE  process  was  selected  for  further  development  in  the 
Analysis  of  Alternatives  Phase.  The  results  of  these  tasks  are  discussed  in  more  detail  in  the 
paragraphs  below.  Last,  a  representative  system  that  had  recently  encountered  problems  during 
OT  was  selected  for  use  during  the  remainder  of  the  project  as  a  baseline  system. 

A.  CREATING  DESIGN  ALTERNATIVES 

1.  Generation  of  Alternative  Solutions 

In  order  to  begin  the  Analysis  of  Alternatives  phase  of  the  project,  the  authors  conducted 
brainstonning  sessions.  The  focus  was  to  come  up  with  ideas  that  could  affect  the  Systems 
Engineering  process,  resulting  in  successful  demonstration  of  interoperability  requirements 
during  operational  testing. 

Each  brainstorming  session  generated  and  discussed  various  ideas  for  solutions,  ensuring 
that  each  member  understood  the  concepts.  Sources  of  ideas  included  the  research  conducted 
during  the  first  phase  and  also  real-world  experiences  of  the  authors.  The  ideas  were  refined 
throughout  various  discussions  and  documented.  While  some  ideas  are  not  as  conventional, 
practicable  or  feasible  as  others;  all  ideas  were  accepted  during  this  process  to  promote  creativity 
and  ensure  as  many  possible  solutions  were  considered  as  possible.  Each  idea  was  assessed  by 
the  group  to  ensure  it  had  the  potential  to  fully  meet  the  requirements  outlined  in  the  previous 
chapters.  As  the  ideas  developed  more  fully,  this  assessment  continued  throughout  the  down 
selection  process,  aiding  in  the  decision  to  eliminate  some  of  the  proposed  idea. 

At  the  completion  of  the  brainstorming,  a  single  list  of  potential  solution  concepts  was 
created.  These  ideas  were  then  reviewed  and  compared.  The  authors  took  this  opportunity  to 
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modify  some  ideas,  combine  similar  ones  and  improve  the  description  of  others  based  on  similar 
ideas  provided.  The  ideas  varied  greatly,  using  various  approaches  that  affected  the  Systems 
Engineering  process  at  different  phases;  however  all  ideas  addressed  methods  to  reduce 
interoperability  failures  in  OT.  Once  these  potential  solutions  were  finalized  and  compiled,  the 
next  step  was  for  the  authors  to  rank  the  relative  effectiveness  and  potential  cost  of  each  solution. 

2.  Initial  Solution  Comparison 

While  most  of  the  suggested  processes  were  worth  considering,  it  was  decided  that  a 
much  smaller  number  of  ideas  should  be  selected  and  refined  for  inclusion  in  the  Analysis  of 
Alternatives.  To  accomplish  this,  the  initial  solutions  were  subsequently  reviewed,  categorized, 
and  consolidated  based  on  two  criteria  factors:  cost  and  effectiveness.  Cost  at  this  step  is  defined 
as  the  additional  financial  burden  of  implementing  and  sustaining  a  potential  solution  in  the 
process.  No  dollar  values  were  calculated;  rather,  the  relative  cost  of  each  solution  was  predicted 
as  compared  to  the  other  solutions.  Similarly  the  relative  effectiveness  or  positive  impact  on  the 
OT&E  pass  rate  was  predicted  for  each  solution  as  compared  to  the  other  proposed  solutions. 

This  initial  evaluation  was  a  qualitative  measure  that  was  used  to  drive  the  decision 
making  process.  With  little  detail  developed  for  each  solution  at  this  point,  the  originator(s)  of 
each  solution  explained  each  potential  solution  and  the  group  discussed  the  benefits  for  each  and 
the  effectiveness  it  would  have  on  decreasing  OT&E  failures.  The  authors  once  again  relied  on 
personal  lessons  learned,  previous  technical  experience,  expert  technical  knowledge,  and 
infonnation  gathered  as  a  part  of  the  research  phase  to  make  these  qualitative  selections.  No 
actual  costs  or  effectiveness  measures  were  generated  at  this  point,  but  rather  the  63  ideas  were 
compared  amongst  each  other,  searching  for  a  reasonable  “bang  for  the  buck.”  It  was 
determined  that  at  such  a  high  level  as  this  initial  comparison,  cost  and  the  effectiveness  of 
reducing  OT  failures  would  be  the  most  important  factors  for  the  stakeholders.  Figure  15  shows 
these  relative  comparisons  plotted  on  two  axes:  Effectiveness  vs.  Cost.  Of  the  63  original  ideas, 
some  were  detennined  to  be  duplicative  and  were  combined  with  the  redundant  counterparts 
while  others  were  detennined  to  be  unachievable  or  unrealistic.  However  their  usefulness  for 
sparking  creativity  and  generating  new  lanes  of  thought  and  ideas  during  the  brainstorming 
process  made  them  relevant  up  until  this  point.  This  resulted  in  41  ideas  being  considered  to  be 
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relevant  and  were  graphed.  A  full  listing  of  the  ideas  that  were  generated  during  the 
brainstorming  sessions  can  be  found  in  Appendix  C. 


Figure  15.  Scatter  Plot  of  Solutions,  Effectiveness  Vs  Cost 

Each  of  the  ideas  was  graphed  based  upon  the  expected  effectiveness  of  each  change  and  the  projected  total 

lifecycle  cost  impact. 


3.  Down  Selection  of  Refined  Solutions 

Once  the  original  list  of  63  ideas  had  been  consolidated  to  4 1  distinct  options  and  these 
options  graphed  for  relative  effectiveness  and  cost,  the  authors  then  selected  a  subset  of  concepts 
that  had  a  high  probability  of  success  and  effectiveness  while  managing  total  ownership  cost. 
Location  on  the  graph  was  not  the  exclusive  selection  criteria;  rather,  it  was  a  means  of  screening 
out  options  that  clearly  did  not  have  a  high  rate  of  return  relative  to  the  expected  implementation 
cost.  After  reviewing  the  4 1  options  individually,  each  author  had  an  opportunity  to  provide 
input  into  the  first  down  selection;  either  from  the  list  or  new  options  conceived  during  the 
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review  process.  There  were  a  total  of  eight  proposals  put  forth  by  various  authors  for  further 
consideration;  these  are  listed  in  Table  4. 

Table  4.  Solutions  for  Improving  Interoperability 

This  table  summarizes  the  down-selected  solutions  that  were  determined  by  the  authors  to  have  a  high  probability’  of 

success  and  manageable  total  ownership  cost. 


Solution  # 

Description 

1. 

Incorporate  backwards  compatibility  in  standards  for  OT 

2. 

DT  exit  criteria  to  include  OT  pass  criteria 

9.  &  10. 

Standardization  of  DT  simulation  and  modeling 

13. 

Use  of  mission  threads  for  interoperability  tests 

15. 

Create  a  new  color  of  money  for  testing 

16. 

Consider  Interoperability  at  all  milestone  reviews 

21. 

Develop  Interoperability  Readiness  Levels  (IORL) 

57. 

Reevaluate  the  passing  criteria  for  interoperability 

As  shown  in  Table  4,  potential  solutions  #9  and  #10  were  combined  as  one  process. 

Most  of  the  processes  selected  have  a  relatively  high  expected  effectiveness  rating  while  the  cost 
ratings  range  from  low  to  high  meaning  that  the  anticipated  cost  to  implement  these  proposed 
improvements  varies  significantly. 

After  reviewing  the  suggested  solutions,  the  authors  then  discussed  each  of  these  options 
to  determine  if  there  was  sufficient  interest  in  pursuing  each  solution  and  to  further  reduce  to  a 
manageable  number  of  solutions  (five  or  less).  To  achieve  this,  a  democratic  process  of 
elimination  was  used.  Each  of  the  authors  chose  three  proposals  they  would  like  to  see 
considered  in  the  AoA  based  on  engineering  judgment.  It  was  quickly  evident  that  five  of  the 
eight  proposals  had  interest  from  at  least  40%  to  90%  of  the  group,  while  the  remaining  three 
were  only  selected  by  one  or  two  of  the  authors.  These  three  with  only  limited  support  were 
eliminated,  and  no  further  reduction  was  required.  Table  5  summarizes  the  final  five  processes 
that  were  selected  for  AoA. 


48 


Table  5.  Final  Alternative  Solutions 

This  table  summarizes  the  final  alternative  solutions  that  were  selected  for  the  Analysis  of  Alternatives. 


Solution  # 

Solution  Description 

Effectiveness 

Cost 

2 

DT  exit  criteria  to  include  OT  pass  criteria 

Very  Effective 

High 

9  &  10 

Standardization  of  DT  simulation  and  modeling 

Very  Effective 

Medium 

13 

Test  to  Mission  Thread  at  interoperability  OT 

Effective 

Medium 

16 

Consider  interoperability  @  all  milestone 
reviews 

Effective 

Low 

21 

Develop  Interoperability  Readiness  Levels 

Very  Effective 

Low 

4.  Refinement  of  the  Alternatives 

At  this  point  the  final  down  selection  could  be  made  in  the  AOA.  While  the  full 
development  would  only  be  completed  on  the  final  alternative  selected,  all  of  the  authors  did  not 
fully  understand  each  other’s  ideas  at  this  point  to  make  a  final  decision.  Therefore  each  of  the 
five  alternatives  was  further  defined,  by  the  originating  author.  This  was  each  author’s 
opportunity  to  advocate  their  option  as  the  best  alternative  to  the  body  of  authors.  The  following 
five  alternative  descriptions  were  used  as  frame  of  reference  to  choosing  the  one  alternative  that 
would  be  developed;  they  were  not  intended  to  be  fully  developed  solutions  and  are  provided 
here  solely  to  illustrate  another  step  in  the  progression  toward  the  final  solution. 

a.  DT Exit  Criteria  to  Include  OT Pass  Criteria 

The  first  alternative  was  focused  on  preventing  programs  from  entering  Operational  Test 
and  Evaluation  (OTE&E)  despite  known  problem  areas  in  the  Critical  Operational  Issues  (COI). 
This  option  utilizes  the  TEMP  to  define  COIs  to  which  the  system  is  tested  to  and  evaluated 
against.  During  the  early  stages  of  development  the  TEMP  has  not  been  fully  developed,  thus 
the  effectiveness  of  this  approach  is  limited  during  early  phases  of  a  program.  However  as  the 
TEMP  is  refined  it  is  possible  to  align  the  DT  exit  criteria  to  the  OT  pass  requirements  by 
including  Operational  Testing  pass  criteria  during  Development  Testing.  OT  and  DT  viewpoints 
are  often  completely  different  as  discussed  in  Chapter  II  of  this  report,  specifically  Table  2; 
however,  this  does  not  mean  they  cannot  work  together.  OT  and  DT  usually  differ  in  what  is 
being  tested,  the  way  tests  are  conducted,  the  test  measurements  and  the  evaluation  criteria.  DT 
generally  involves  testing  using  a  black  box  concept  or  a  component  as  a  part  of  the  basic  system 
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with  little  to  no  testing  of  the  complete  system;  in  contrast  to  OT  which  is  always  at  the  system 
level.  Under  this  concept,  the  final  testing  in  DT  would  include  testing  of  the  complete  system. 

The  current  process  for  DT  only  tests  technical  criteria  under  a  controlled  environment. 
DT  should  be  able  to  measure  the  quality  of  a  basic  combat  condition.  The  current  process 
measures  particular  parameters  (i.e.  the  number  of  g’s  pulled  as  the  missile  acquires;  missile 
launch  velocity;  etc...).  In  addition  to  technical  specifications,  DT  should  test  performance 
capabilities  (continuing  with  the  same  missile  example,  the  successful  tracking  of  the  target  by 
radar,  etc).  DT  is  performed  by  the  Development  Contractor  and  can  be  witnessed  by 
Independent  Verification  and  Validations  (IV&V),  and/or  any  other  designated  representatives. 
Exit  criteria  or  successful  completion  of  DT  testing  requires  that  all  the  test  documentation  has 
been  approved,  all  test  scripts  have  been  executed  and  trouble  reports  are  generated  for  each 
failure  or  anomaly,  all  trouble  reports  have  been  resolved  all  changes  made  as  a  result  of  trouble 
reports  have  been  tested,  and  the  test  report  has  been  reviewed  and  approved.  DT  is  mainly 
conducted  at  the  development  contractor’s  facility  and  usually  does  not  have  physical  interface 
with  external  systems.  By  including  OT  pass  criteria  to  DT  exit  criteria;  it  will  require  early 
involvement  of  the  Operational  Test  Community  (OPTEVFOR)  to  ensure  the  interoperability 
requirements  are  included  in  the  DT&E  process. 

For  this  solution,  the  development  agent  and  OPTEVFOR  will  work  closely  to  develop 
the  T&E  Master  plan  at  the  early  program  development  cycle.  The  Critical  Operational  Issues 
(COIs)  that  relate  to  operational  effectiveness  and  operational  suitability  need  to  be  defined  and 
examined  to  determine  the  system’s  capability  to  perfonn  its  mission.  The  system  integrator  will 
implement  and  incorporate  the  COIs  into  the  development  test  (DT)  plan.  If  there  are 
interoperability  issues  identified  during  the  DT,  the  issues  need  to  be  documented  and  changes  to 
system  detailed  design  may  be  required.  A  simple  functional  flow  block  diagram  of  this  process 
is  shown  in  Figure  16. 
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Figure  16.  Basic  Process  Diagram  for  the  “DT  Exit  Criteria  to  Include  OT  Pass  Criteria”  Option 

Functional  Flow  Block  Diagram  of  the  “Exit  Criteria  to  Include  OT  Pass  Criteria  ”  Alternative. 

Implementing  OT  passing  criteria  into  DT  will  increase  the  cost  of  the  program  in  order 
to  conduct  the  basic  operational  test.  This  would  also  require  additional  personnel,  training  and 
test  equipment  (hardware  and  software),  as  well  as  a  considerable  increase  in  coordination  with 
other  systems  or  organizational  entities.  This  increases  both  the  DT  schedule  and  cost,  while 
potentially  decreasing  the  amount  of  money  and  time  spent  at  OT.  However,  by  preventing  re¬ 
work  and  re-testing  and  delivering  a  quality  product  when  scheduled,  this  concept  could 
potentially  save  money  and  time  over  the  life  of  the  program. 

Program  schedules  would  be  affected  by  implementing  these  changes  in  the  process. 
However,  these  schedules  could  be  affected  even  worse  if  the  product  doesn’t  meet  the  basic 
requirements.  If  the  process  is  included  in  the  schedule  from  the  beginning  of  the  project,  the 
overall  impact  would  likely  be  minimal.  An  advantage  would  be  that  the  longer  schedule  would 
be  known  from  the  beginning  and  unexpected  delays  due  to  OT  failure  would  be  significantly 
less  likely. 

Implementation  of  this  change  would  be  difficult.  In  order  to  test  in  a  basic  operational 
environment  it  would  require  coordination  with  other  systems.  Deciding  who  is  paying  what 
would  make  the  implementation  difficult.  Other  systems  schedules  would  need  to  be  taken  into 
consideration. 

Implementing  this  process  would  have  a  positive  impact  on  the  final  product.  It  would 
improve  the  OT  schedule  and  success.  The  final  product  would  be  more  reliable  and  less  likely 
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to  encounter  delays  as  a  result  of  failing  OT.  This  would  lead  to  cost  savings  and  product 
satisfaction. 

b.  Standardization  of  DT  Modeling  and  Simulation 

The  standardization  of  Developmental  Testing  (DT)  and  Modeling  and  Simulation 
(M&S)  is  a  second  possible  solution  to  this  problem.  M&S  should  be  used  during  DT  as  a 
method  for  predicting  system  perfonnance,  identifying  technology  and  performance  risk  areas, 
and  the  ability  to  support  determining  system  effectiveness  and  suitability.  Currently,  with  DoD 
programs  failing  to  detect  critical  interoperability  failures  prior  to  operational  level  testing  after 
successfully  completing  DT,  there  is  a  need  for  standardization  of  DT  and  M&S  procedures  and 
pass  criteria.  The  use  of  quality  M&S  within  the  DT  process  could  greatly  enhance  the 
achievement  of  effective  and  efficient  test  and  evaluation  of  the  system  all  the  way  through  to 
OT&E.  M&S  is  currently  used  within  programs  to  demonstrate  system  integration  risks, 
supplement  live  testing  with  M&S  stressing  the  system,  assist  in  planning  the  scope  of  live  tests 
and  data  analysis,  and  prediction  of  system  performance  and  possible  risk  areas.  None  of  the 
current  M&S  strategies  are  standardized  and  most  model  and  simulations  are  tailored  for  their 
specific  system,  but  the  following  list  from  the  Defense  Acquisition  Guidebook  [DAU,  2010] 
contains  some  areas  that  could  be  considered  for  an  overall  standardized  M&S  method  that  leads 
to  a  successful  DT: 

•  Document  the  intended  use  of  models  and  simulations 

•  Describe  the  data  standards  and  verification,  validation,  and  accreditation  standards  with 
which  the  models  and  simulations  and  associated  data  must  comply 

•  Identify  the  modeling  and  simulation  data  needed  to  support  T&E 

•  Define  the  methodology  for  establishing  confidence  in  the  results  of  models  and 
simulations 

•  Provide  descriptive  information  for  each  model  and  simulation  resource:  Title,  acronym, 
version,  date,  support  organization  (the  organization  with  primary  responsibility  for  the 
model  or  simulation), 

•  State  assumptions,  capabilities,  limitations,  risks,  and  impacts  for  the  models  and 
simulation 

•  Project  the  schedule  and  availability  of  M&S  for  T&E  support 
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M&S  capability  encompasses  many  software  tools,  middleware,  networks,  security 
features  and  encryption  capabilities  that  will  have  to  be  standardized  to  some  degree.  Without 
standardization,  the  credibility  of  M&S  becomes  an  issue.  If  M&S  cannot  provide  trustworthy 
insights  into  the  system  then  the  program  will  be  ill-served  and  the  M&S  investment  will  be 
wasted. 

Costs  and  the  development  schedule  are  two  other  areas  of  consideration  that  need  to  be 
examined  when  looking  into  DT  and  M&S  standardization.  The  associated  schedule  and  costs 
with  implementing  a  comprehensive  set  of  standards  for  M&S  within  the  DT  cycle  could  prove 
to  be  extensive.  Estimates  could  be  detennined  from  previous  standards  and  process 
implementations  as  well  as  the  projected  time  taken  to  complete  those  processes.  Putting 
standards  in  place  for  M&S  and  DT  can  have  lasting  effects  for  system  integration  and  system 
interoperability.  The  potential  for  cost  saving  during  the  test  cycle  can  best  be  captured 
qualitatively.  M&S  standardization  will  trim  down  DT  cost  and  schedule  by  introducing 
complete  and  proven  models  and  simulations,  however  it  is  likely  that  not  all  aspects  of  the 
standard  will  be  applicable  to  every  system  and  therefore  some  unnecessary  costs  will  be 
incurred.  If  standardized  M&S  methods  are  used  across  all  programs  this  could  lead  to  cost  and 
schedule  savings,  however  time  and  funding  will  be  required  to  develop  and  validate  the 
standards  prior  to  implementation. 

To  improve  the  systems  engineering  process,  M&S  enables  better  planning  of  live  test 
events,  representing  system  attributes  that  cannot  be  examined  realistically  in  live  testing. 
However,  without  standardizing  the  M&S  process,  the  system  may  suffer  increase  of  cost, 
inadequate  discipline  in  planning  or  applying  M&S,  incoherent  plan  and  no  information  sharing 
among  community. 

To  achieve  a  successful  DT  which  eventually  leads  to  a  successful  OT,  the  M&S  process 
should  be  standardized  in  the  following  areas: 

•  Document  the  intended  used  of  models  and  simulation 

•  Standardized  Verification,  Validation,  and  Accreditation  (VV&A)  process  to  ensure 
development  of  correct  and  valid  simulation 

•  Identify  the  M&S  data  needed  to  support  DT 

•  Define  the  methodology  for  establishing  confidence  in  the  results  of  models  and 
simulations 
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•  Provide  descriptive  information  for  each  model  and  simulation  resource 

•  State  assumptions,  capabilities,  limitations,  risks,  and  impacts  for  the  models  and 
simulation 

•  Project  the  schedule  and  availability  for  DT  support 

A  simple  functional  flow  block  diagram  of  this  M&S  process  is  shown  in  Figure  17. 


Figure  17.  Basic  Process  Diagram  for  the  “Standardization  of  DT  Modeling  and  Simulation” 

Option 

Functional  Flow  Block  Diagram  of  the  “Standardization  of  DT  Modeling  and  Simulation  ”  Alternative. 

Implementation  of  standard  M&S  methods  and  processes  during  DT  will  be  a  moderately 
difficult  task  to  accomplish.  New  and  existing  methods  will  need  to  be  blended  to  ensure 
flexibility  and  completeness  are  captured  in  the  M&S  process  to  make  certain  all  programs  will 
have  confidence  in  the  new  standardized  approach.  New  and  current  M&S  documentation  should 
be  modified  and  streamlined  for  concise  understanding  of  how  M&S  will  integrate  with  the  DT 
cycle.  Further,  there  is  always  risk  involved  with  developing  standards.  As  standards  are 
rewritten  they  are  hardly  ever  loosened,  but  rather  tend  to  become  more  and  more  stringent  with 
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time.  This  could  eventually  lead  to  time  and  money  wasted  testing  to  standards  that  have 
become  unrealistic  (outside  the  mission  thread  requirement)  or  archaic  (outdated  and  irrelevant). 

c.  Test  to  Mission  Thread 

The  tenn  mission  critical  thread  is  being  used  more  and  more  often  throughout  the  DOD. 
The  Joint  Chiefs  of  Staff  Instruction  (CJCSI)  62 12.0  IE  states:  “The  joint  critical  threads  are 
detennined  by  the  sponsor’s  analysis  of  the  system’s  required  operational  capabilities  and  other 
Key  Perfonnance  Parameters.”  The  CJCSI  62 12.0  IE  goes  on  to  give  a  formal  definition  of  a 
Joint  Mission  Critical  Thread  as  “an  operational  and  technical  description  of  the  end  to  end  set  of 
activities  and  systems  that  accomplish  the  execution  of  a  joint  mission”  [CJCSI  6212. 01E,  2008]. 
For  the  purposes  of  developing  this  option  the  authors  have  chosen  to  use  this  CJCSI  62 12.0  IE 
definition  of  a  mission  critical  thread. 

Mission  threads  should  be  developed  early  in  the  acquisition  process.  Mission  threads 
could  be  taken  from  a  previously  developed  list  or  developed  specifically  for  the  system  under 
development;  either  way  the  mission  threads  must  reflect  the  operational  requirements  of  the 
system.  These  mission  threads  must  be  designed  specifically  to  quantify  a  system’s  response  to  a 
series  of  scenarios.  These  threads  can  then  be  applied  as  a  guide  for  building  the  system 
architecture,  where  it  can  help  identify  the  architectural  risk  early  in  the  process.  When  applied 
to  the  SoS,  mission  threads  can  be  used  as  inputs  for  building  the  SoS  architecture,  and  can  also 
serve  as  a  tool  for  system  in-progress  design  review  and  system  architecture  evaluation.  The 
mission  threads  will  need  to  be  applied  to  the  interoperability  tests  perfonned  during  system 
development.  As  acquisition  progresses,  mission  threads  will  be  applied  to  testing,  gradually 
increasing  the  number  of  systems  involved.  While  it  may  push  the  schedule  to  the  right,  it  will 
greatly  reduce  the  risk  of  severe  delay  due  to  integration  issues. 

As  the  system  matures  and  begins  to  interact  with  other  systems  the  mission  thread 
should  be  reevaluated  to  encompass  the  mission  threads  of  these  respective  systems.  In  this 
manner  the  system  and  its  mission  thread  evolve  to  encompass  a  larger  and  larger  view  with  the 
last  capturing  the  SoS  prior  to  entering  OT.  Figure  18  shows  a  functional  flow  block  diagram  of 
this  process. 
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Figure  18.  Basic  Process  Diagram  for  the  “Test  to  Mission  Thread”  Option 

Functional  Flow  Block  Diagram  of  the  “Test  to  Mission  Thread’’  Alternative. 

The  biggest  challenge  facing  the  implementation  of  mission  threads  is  the  fact  that  each 
system  is  normally  operated  and  managed  independently.  Each  system  has  its  own  program 
office  responsible  for  its  own  system  development.  However,  each  system  may  interact  with 
different  joint  programs,  resulting  in  a  complex  combination  of  systems.  The  combination  of  the 
multiple  mission  threads  must  be  tested  during  OT  in  order  to  have  interoperability  for  the  entire 
SoS  evaluated.  The  mission  threads  test  will  be  required  at  a  system  level  and  would  need  to  be 
started  as  early  as  the  architecture  design  phase.  Then  as  the  integrated  system  matures,  the 
combined  mission  threads  will  be  verified  through  simulations  or  tabletop  exercises  at  each 
milestone.  Failures  will  need  to  be  corrected  before  moving  onto  the  next  stage  of  the  process. 
An  example  of  a  mission  thread  for  a  combat  system  might  be  to  engage  (acquire,  track,  and 
destroy)  10  incoming  threats,  each  of  these  threats  would  be  defined  in  detail  as  part  of  the 
mission  thread.  This  high  level  approach  is  much  different  from  a  technical  requirement  such  as 
display  200  simultaneous  air  targets  or  process  data  at  a  rate  of  60Mb/sec.  Completing  the 
mission  critical  thread  is  based  on  mission  completion,  not  on  meeting  a  system  specification. 

The  cost  and  schedule  of  the  program  may  increase  when  mission  thread  testing  is 
involved  in  the  early  phase  of  the  process.  However,  it  avoids  the  risk  of  an  even  greater  impact 
to  cost  and  schedule  later  in  the  program.  The  problems  that  were  not  detected  until  systems 
integration  or  deployment  will  cause  significant  issues  in  cost,  schedule,  and  perfonnance. 
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d.  Consider  Interoperability  at  Milestone  Reviews 

It  is  understandable  that  most  systems  are  used  as  part  of  a  larger  system.  It  is  important 
that  each  system  can  be  fully  integrated  into  a  system  of  systems  (SOS).  A  successful 
integration  of  systems  involves  the  perfonnance  of  all  systems  working  together  to  achieve  the 
desired  tasks.  The  integration  requires  that  each  system  inter-operate  with  one  another  to  meet 
the  designed  capability  of  the  system  of  systems.  The  interoperability  of  systems  must  be 
considered  early  and  applied  continuously  across  the  system  of  systems  life  cycle. 

Considering  interoperability  at  each  milestone  review  is  one  such  way  to  ensure  that  all 
systems  work  together  to  achieve  the  desired  tasks.  It  allows  system  engineers  to  periodically 
evaluate  the  interoperability  of  the  system.  This  evaluation  is  carried  out  as  part  of  the  milestone 
reviews.  The  generalized  process  will  need  to  be  tailored  for  each  program  and  each  review. 

For  instance,  during  Milestone  A  much  of  the  consideration  will  be  based  on  the  conceptual 
design  of  the  system,  while  at  Milestone  C  interoperability  will  be  assessed  based  on  the  results 
of  testing.  The  general  process  for  evaluating  interoperability  consists  of: 

•  assess  the  system 

•  evaluate  interoperability  results 

•  take  appropriate  action 

•  proceed  to  the  next  milestone 

During  the  milestone  review,  a  panel  of  experts  will  assess  the  system  to  detennine 
systems,  components,  and  sub  components  to  evaluate  for  interoperability.  They  will  then  assess 
the  interoperability  of  these  components  and  identify  any  that  have  a  low  interoperability.  The 
program  manager  will  then  be  responsible  for  addressing  any  identified  interoperability  issues, 
such  as  those  demonstrating  a  low  interoperability. 


57 


Figure  19.  Basic  Process  Diagram  for  the  “Consider  Interoperability  at  Milestone  Reviews” 

Option 

Functional  Flow  Block  Diagram  of  the  "Consider  Interoperability  at  Milestone  Review’s  ”  alternative. 

By  considering  interoperability  at  each  milestone  review,  one  can  gain  a  sense  of  the 
level  of  interoperability  of  the  system.  This  will  allow  the  program  manager  to  act  accordingly. 
Allowing  for  the  detection  of  low  levels  of  interoperability  at  each  of  the  milestones  enables 
systems  to  avoid  the  high  costs  of  failing  OT.  Thus  considering  interoperability  at  each 
milestone  review  has  the  potential  to  have  significant  cost  savings. 

It  would  be  relatively  easy  to  implement  a  process  to  consider  interoperability  at  each 
milestone  review  given  the  fact  that  subject  matter  experts  are  on  hand  during  the  milestone 
reviews.  As  such  this  approach  would  be  cost  effective  to  implement  and  moderately  effective  at 
preventing  OT  failures. 

e.  Define  and  Apply  Interoperability  Readiness  Levels  (I/ORLs) 

The  last  potential  solution  would  be  to  develop  and  implement  a  system  of 
Interoperability  Readiness  Levels  (I/ORLs)  to  quantify  the  maturity  of  the  system  interfaces. 
Implementing  I/ORLs  is  a  way  to  use  metrics  to  assess  the  maturity  of  a  system's  interoperability 
before  proceeding  to  Operational  Testing  (OT).  Using  metrics  allows  for  a  quantitative 
judgment  of  a  system's  readiness  for  OT,  removing  some  of  the  subjectivity  of  humans.  This 
allows  for  more  consistency  in  detennining  when  a  system  is  ready  for  OT. 

Assessing  a  system  with  differing  pieces  of  integrated  technology  based  on  I/ORLs 
reflects  an  evaluation  of  integration  and  readiness;  it  gauges  the  maturity  of  the  interfaces  or 
interoperability  of  systems.  I/ORLs  can  be  applied  to  connections  between  legacy  systems, 
legacy  and  new  systems,  or  between  integration  of  new  systems.  Each  interface  will  have  a 
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unique  I/ORL  value,  thus  it  is  possible  for  a  system  to  be  assigned  multiple  I/ORLs.  Further, 
assessing  a  system  based  on  I/ORLs  can  be  perfonned  on  technology  insertion  or  on  new  system 
development. 

To  gain  a  frame  of  reference  for  the  creation  of  interoperability  readiness  levels  (I/ORLs), 
the  authors  started  with  open  literature  research.  This  research  yielded  several  articles  on  the 
subject  of  integration  readiness  levels  (IRLs).  The  reference  frame  provided  in  A  Systems 
Approach  to  Expanding  the  Technology  Readiness  Level  within  Defense  Acquisition  [Sauser,  et 
al,  2009]  provides  a  conceptual  basis  from  which  the  authors  could  develop  IRLs.  The 
integration  readiness  levels  summarized  in  Table  6  illustrate  the  basic  concept  of  an  Integration 
Readiness  Level  system  which  is  similar  to  an  Interoperability  Readiness  Level  system.  If  this 
option  is  selected  a  more  appropriate  set  of  I/ORLs  would  be  developed  by  the  authors  after  the 
AOA  is  complete.  In  practice  a  team  would  assess  the  system  based  on  a  criteria  of  agreed  upon 
levels,  if  the  system  scored  at  a  level  below  the  agreed  upon  level,  the  system  would  not  proceed 
to  OT  but  rather  continue  further  development. 

Table  6.  An  Example  of  Integration  Readiness  Levels  (IRLs) 

This  table  summarizes  the  Integration  Readiness  Levels  that  were  identified  by  Sauser,  2009. 


IRL 

Definition 

Description 

9 

Integration  is  Mission  Proven 
through  successful  mission 
operations. 

IRL  9  represents  the  integrated  technologies 
being  used  in  the  system  environment 
successfully.  In  order  for  a  technology  to  move  to 
the  TRL  9,  it  must  first  be  integrated  into  the 
system  and  then  proven  in  the  relevant 
environment;  thus,  progressing  IRL  to  9  also 
implies  maturing  the  component  technology  to 
the  TRL  9. 

8 

Actual  integration  completed 
and  Mission  Qualified  through 
test  and  demonstration  in  the 
system  environment. 

IRL  8  represents  not  only  the  integration-meeting 
requirements,  but  also  a  system-level 
demonstration  in  the  relevant  environment.  This 
will  reveal  any  unknown  bugs/defects  that  could 
not  be  discovered  until  the  interaction  of  the  two 
integrating  technologies  was  observed  in  the 
system  environment. 

7 

The  integration  of  technologies 
has  been  Verified  and 

Validated  with  sufficient  detail 
to  be  actionable. 

IRL  7  represents  a  significant  step  beyond  IRL  6; 
the  integration  has  to  work  from  a  technical 
perspective,  but  also  from  a  requirements 
perspective.  IRL  7  represents  the  integration 
meeting  requirements  such  as  performance, 
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IRL 

Definition 

Description 

throughput,  and  reliability. 

6 

The  integrating  technologies  can 

Accept,  Translate,  and 
Structure  Information  for  its 

intended  application. 

IRL  6  is  the  highest  technical  level  to  be 
achieved;  it  includes  the  ability  to  not  only 
control  integration,  but  to  specify  what 
infonnation  to  exchange,  to  label  units  of 
measure  to  specify  what  the  infonnation  is,  and 
the  ability  to  translate  from  a  foreign  data 
structure  to  a  local  one. 

5 

There  is  sufficient  Control 
between  technologies  necessary 
to  establish,  manage,  and 
tenninate  the  integration. 

IRL  5  simply  denotes  the  ability  of  one  or  more 
of  the  integrating  technologies  to  control  the 
integration  itself;  this  includes  establishing, 
maintaining,  and  tenninating. 

4 

There  is  sufficient  detail  in  the 
Quality  and  Assurance  of  the 
integration  between 
technologies. 

IRL  4  Many  technology-integration  failures  never 
progress  past  IRL  3,  due  to  the  assumption  that  if 
two  technologies  can  exchange  infonnation 
successfully,  then  they  are  fully  integrated.  IRL  4 
goes  beyond  simple  data  exchange  and  requires 
that  the  data  sent  is  the  data  received  and  there 
exists  a  mechanism  for  checking  it. 

3 

There  is  Compatibility  (i.e., 
common  language)  between 
technologies  to  orderly  and 
efficiently  integrate  and  interact. 

IRL  3  represents  the  minimum  required  level  to 
provide  successful  integration.  This  means  that 
the  two  technologies  are  able  to  not  only 
influence  each  other,  but  also  to  communicate 
interpretable  data.  IRL  3  represents  the  first 
tangible  step  in  the  maturity  process. 

2 

There  is  some  level  of 
specificity  to  characterize  the 
Interaction  (i.e.,  ability  to 
influence)  between  technologies 
through  their  interface. 

IRL  2  Once  a  medium  has  been  defined,  a 
“signaling”  method  must  be  selected  such  that 
two  integrating  technologies  are  able  to  influence 
each  other  over  that  medium.  Since  IRL  2 
represents  the  ability  of  two  technologies  to 
influence  each  other  over  a  given  medium,  this 
represents  integration  proof-of-concept. 

1 

An  Interface  between 
technologies  has  been  identified 
with  sufficient  detail  to  allow 
characterization  of  the 
relationship. 

IRL  1  This  is  the  lowest  level  of  integration 
readiness  and  describes  the  selection  of  a  medium 
for  integration. 

To  conduct  the  recommended  assessment,  one  must  select  a  review  at  which  the 
assessment  will  take  place.  This  review  may  be  conducted  a  single  time;  however  it  is 
recommend  that  multiple  reviews  be  conducted.  Regardless  of  the  number  of  reviews  selected 
the  process  will  be  the  same  each  time.  First  a  review  is  selected  in  which  the  necessary  subject 
matter  experts  are  available.  The  system  is  then  assessed  to  detennine  its  I/ORL  value  in  a 
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manner  similar  to  that  presented  in  Table  6.  These  I/ORL  ratings  are  then  evaluated  to  determine 
if  the  system  is  progressing  at  an  acceptable  rate  with  respect  to  interoperability.  Based  on  the 
results  of  this  evaluation  appropriate  action  may  be  taken  to  improve  interoperability 
functionality.  If  multiple  reviews  are  conducted,  relative  progress  may  be  tracked.  To  gain  a 
visual  understanding  of  these  high  level  functions,  a  top  level  functional  flow  block  diagram  has 
been  provided  in  Figure  20.  The  detailed  definitions  of  an  I/ORL  will  be  provided  in  Chapter  IV 
of  this  report. 


Figure  20.  Process  Diagram  for  “Define  and  Apply  I/ORLs”  Option 

Functional  Flow  Block  Diagram  of  the  “Interoperability’  Readiness  Levels  ”  alternative. 

Implementing  I/ORLs  on  a  system  is  an  assessment  of  the  system.  It  does  not  require 
development  or  testing.  As  such,  the  cost  to  implement  this  assessment  is  small.  The  impact  to 
cost  is  based  on  the  time  it  takes  to  evaluate  the  system  against  a  set  of  I/ORLs  and  the 
administrative  cost  to  document  this  evaluation.  During  the  course  of  an  acquisition  cycle 
stakeholders  and  subject  matter  experts  (SMEs)  meet  on  a  regular  basis  for  development  and 
other  assessments.  Assessing  a  system  for  interoperability  can  be  performed  during  any  or 
several  of  these  existing  meetings  rendering  the  cost  minimal.  Implementing  I/ORLs  can 
potentially  save  a  significant  amount  of  money.  By  preventing  a  system  with  immature 
interfaces  from  proceeding  to  OT  and  failing,  money  spent  on  unnecessarily  testing  an  immature 
system  could  be  saved. 
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Implementing  I/ORLs,  can  impact  schedule  but  the  impact  can  be  minimal.  As  noted,  the 
I/ORL  assessment  can  be  implemented  during  one  of  the  many  existing  pre -Milestone  C 
meetings.  The  expected  impact  on  the  schedule  would  be  one  or  two  days  per  meeting.  If  these 
few  days  save  the  time  it  takes  for  the  system  to  go  through  OT  testing  and  fail,  the  result  could 
be  a  savings  in  schedule. 

Implementing  this  change,  functionally,  is  not  difficult.  Systems  are  routinely  assessed 
during  development.  I/ORLs  can  be  one  additional  standard  by  which  to  measure  a  system.  The 
I/ORL  assessment  can  be  performed  during  one  or  many  of  pre-Milestone  C  events.  It  can  be 
perfonned  once  or  several  times  to  see  a  progression  of  interoperability  maturity.  One  event  in 
particular  in  which  an  I/ORL  assessment  can  be  performed  is  during  a  Post-Critical  Design 
Review  Assessment  which  ends  Integrated  System  Design  during  Milestone  B. 

Implementing  I/ORLs  can  improve  OT  success.  Assessing  a  system  based  on  LORLs 
can  prevent  the  system  which  is  not  sufficiently  interoperable  from  proceeding  to  OT,  saving  the 
system  from  failing.  This  assessment  can  inform  on  specific  areas  of  development  needing 
further  development.  In  the  end  this  would  make  a  system  having  undergone  an  I/ORL 
assessment  more  stable  and  dependable. 

B.  PUGH  MATRIX 

The  method  selected  for  down  selecting  from  these  five  alternatives  to  a  single  proposed 
solution  was  a  Pugh  Matrix.  Evaluation  criteria  were  developed  based  on  the  process 
requirements  developed  in  the  previous  phase,  the  authors  evaluated  each  alternative  against 
these  criteria  and  the  results  of  the  individual  Pugh  Matrix  responses  were  tabulated.  The  results 
of  the  Pugh  Matrix  were  discussed  and  validated  by  the  authors  and  were  used  as  the  major  tool 
for  the  group  selection  of  the  final  alternative  to  the  current  SE  process  that  would  be  developed 
in  detail  during  the  third  phase  of  the  project 

1.  Pugh  Matrix  Comparison  Criteria 

Concurrent  with  the  refinement  of  the  five  alternatives  discussed  previously,  the  authors 
developed  the  evaluation  criteria  for  assessing  and  ranking  these  five  alternatives.  The  simple 
cost  versus  effectiveness  comparison  was  detennined  to  be  insufficient.  Nine  criteria  were 
developed  in  all  based  upon  the  requirements  and  value  system  that  were  developed  in  the 
previous  phase  and  documented  in  Chapter  II  of  this  report.  Effectiveness  was  divided  into  the 
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increase  in  the  percent  of  OT  passed  as  well  as  how  early  in  the  program  an  issue  was  likely  to 
be  detected  with  the  given  alternative.  Additionally,  cost  was  analyzed  more  closely  and 
encompassed  not  just  financial  cost  but  also  schedule  cost  and  risk.  These  costs  were  also 
addressed  both  as  upfront  cost  for  implementing  the  new  system  and  the  reoccurring  costs  that 
would  negatively  or  positively  impact  each  program. 

The  nine  criteria: 

•  Change  in  OT  %  Pass:  This  rates  each  alternative’s  increased  or  decreased  chance  of 
passing  OT  over  implementing  Alternative  1 . 

•  Change  in  the  program  phase  in  which  problem  detection  occurs:  This  rates  when  each 
alternative  allows  the  detection  of  issues  prior  to  OT.  How  much  sooner  are  those 
problems  predicted  and  mitigated  or  avoided  compared  to  Alternative  1? 

•  Upfront  cost  of  implementing  the  new  process  (upfront  $):  This  is  a  comparison  of  the 
estimated  costs  to  develop  the  new  SE  process  compared  to  Alternative  1 .  Exact  cost  is 
not  required,  this  is  a  relative  comparison.  Is  the  alternative  to  be  more  or  less  expensive 
to  implement  that  Alternative  1?  This  is  the  one-time  financial  costs  for  starting  the  new 
process. 

•  Per  program  (reoccurring)  costs  for  the  new  process  (per  program  $):  This  is  a 
comparison  of  the  estimated  costs  of  using  the  new  SE  process  compared  to  Alternative  1. 
Exact  costs  are  not  required,  this  is  a  relative  comparison.  Is  the  alternative  likely  to  be 
more  or  less  expensive  for  each  project  to  use  then  Alternative  1?  This  is  the  recurring 
costs  of  the  new  process  that  will  affect  each  program. 

•  Approval  Level  (getting  the  new  process  approved  for  DOD  use)  (upfront  risk  and  time): 
This  is  the  comparison  of  each  alternative  with  Alternative  1  ’s  time  and  risk  of  getting 
approval  for  the  new  process.  For  example,  one  alternative  might  be  at  the  USD  (AT&L) 
while  another  may  only  be  at  the  NSWC  CO  level.  (Approval  of  the  new  process  is  only 
needed  once,  not  for  each  program.) 

•  Coordination  required  between  agencies  to  implement  the  new  process  (upfront  time): 
This  is  a  comparison  of  the  one-time  “time”  expense  required  for  developing  the  new 
processes  (Alternative  compared  to  Alternative  1 .) 

•  Time  to  develop,  train,  and  implement  the  new  process  (upfront  time  and  money):  This 
is  a  comparison  of  the  one-time  “time”  and  financial  cost  of  developing  training  and 


63 


transition  to  the  new  process  across  program  offices  (complexity  of  change  required  for 
the  alternative  compared  to  Alternative  1 .) 

•  Impact  to  individual  program  schedules  (per  program,  time):  Comparison  of  how  the 
alternative  affects  each  program  schedule  compared  to  the  effects  of  Alternative  1. 

•  Change  in  documentation  for  each  program  (per  program  time  and  $):  Comparison  of  the 
increase  or  decrease  in  time  and  financial  cost  of  documentation  of  the  alternative 
compared  to  Alternative  1 . 

2.  Pugh  Matrix  Data  Acquisition 

Figure  2 1  is  a  representative  sample  of  the  Pugh  Matrix  completed  by  each  member  of 
the  team;  Table  7  is  the  legend  used  by  the  team  members  in  developing  their  Pugh  Matrices. 
The  authors  choose  to  do  individual  Pugh  Matrices  so  that  each  group  member  had  an  equal  say 
in  the  further  development  of  the  project. 
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Change  in  OT  %  Pass 

0 

0 

Change  in  the  program  phase  in  which  problem 
detection  occurs 

0 

+ 

+ 

0 

Upfront  cost  of  implementing  the  new  process 
(upfront  $) 

0 

+ 

0 

Per  program  (reoccuring)  costs  for  the  new 
process  (per  program  $) 

0 

+  + 

+  + 

+  + 

+ 

Approval  Level  (getting  the  new  process  approved 
for  DOD  use)  (upfront  risk  and  time) 

0 

+ 

Coordination  required  between  agencies  to 
implement  the  new  process  (upfront  time) 

0 

0 

Time  to  develop  train  and  implement  of  the  new 
process  (upfront  time) 

0 

Impact  to  individual  program  schedules  (per 
program,  time) 

0 

+  + 

+  + 

+  + 

+ 

Change  in  documentation  for  each  program  (per 
program  time  and  $) 

0 

0 

0 

0 

0 

Figure  21.  Sample  Pugh  Matrix  from  one  of  the  fourteen  group  members. 


The  authors  evaluated  each  option  using  a  Pugh  Matrix  and  the  nine  criteria  identified  above. 
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Table  7.  Pugh  Matrix  Legend 

Each  option  in  the  AoA  was  evaluated  for  each  of  the  nine  criteria  based  on  these  rating  criteria. 


Definition 

+  + 

Major  improvement  compared  to  Alternative  1(  Include  OT  pass  criteria  in  DT  exit) 
(significant  decrease  in  time,  cost,  performance  or  risk) 

+ 

Some  improvement  compared  to  Alternative  1(  Include  OT  pass  criteria  in  DT  exit)(some 
decrease  in  time,  cost,  performance  or  risk) 

0 

No  difference  from  current  process  (doing  nothing) 

Some  increased  burden/loss  compared  to  Alternative  1  (  Include  OT  pass  criteria  in  DT 
exit)(some  increase  in  time,  cost,  performance  or  risk) 

-  - 

Major  increased  burden/loss  compared  to  Alternative  1(  Include  OT  pass  criteria  in  DT 
exit)(significant  increase  in  time,  cost,  performance  or  risk) 

Since  it  is  assumed  that  our  alternative  will  be  an  improvement  over  the  current  system, 
there  was  no  “do  nothing”  alternative.  This  is  important  because  the  use  of  a  “do  nothing” 
alternative  will  always  score  higher  than  any  changes  when  it  comes  to  any  upfront 
developmental  costs  (there  is  no  cost  for  making  no  change)  and  may  often  have  some  reduced 
reoccurring  financial,  schedule  and  risk  costs  within  each  program.  This  could  give  the  “do 
nothing”  alternative  a  favorable  outcome  in  any  AOA.  However,  the  problem  being  addressed  is 
rarely  solved  by  “doing  nothing”  and  an  AOA  is  generally  not  performed  unless  a  change  is 
thought  to  be  necessary.  Therefore  DT  Exit  Criteria  to  include  OT  Pass  Criteria  (Alternative  1) 
was  chosen  as  the  baseline  alternative.  The  other  four  alternatives  were  compared  to  this  and 
each  group  member  used  their  engineering  and  project  management  judgment  to  detennine  if  the 
other  alternatives  were  “better”  or  “worse”  then  Alternative  1 . 

3.  Pugh  Matrix  Analysis 

With  each  member  completing  their  own  Pugh  Matrix,  the  total  +,  0,  and  -  were  summed 
in  Figure  22,  Pugh  Matrix  Results.  When  analyzed,  there  is  no  single  criterion  that  stands  out  as 
being  irrelevant;  had  any  one  criterion  been  equal  or  nearly  equal  across  the  five  alternatives  then 
considering  that  particular  criterion  would  not  have  provided  any  substantial  decision  making 
capability  when  choosing  between  the  alternatives.  It  was  quickly  noted  however,  that  the  two 
highest  ranked  alternatives  were  closely  related.  That  is  to  say  that  the  highest  rated  alternative, 
“Define  and  Apply  I/ORLs,”  could  possibly  fulfill  the  second  highest  ranked  alternative. 

Further,  each  system  was  analyzed  with  all  14  group  members’  data,  as  well  as  without  the 
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highest  and  lowest  score  provided  to  each  system  by  any  one  group  member  (middle  12  group 
responses).  Similar  to  judging  at  Olympics  and  other  competitions,  this  method  allowed  the 
group  to  reduce  any  biases  that  may  arise  from  a  member  who  may  have  authored  or  been 
endeared  to  any  one  alternative.  No  significant  bias  appeared  to  be  present,  as  the  score  of  each 
alternative  changed  by  no  more  than  four  +  or  -,  out  of  a  possible  fluctuation  of  up  to  36  +  or  -. 


Delta  in  OT  %  Pass 

0 

-2 

11 

5 

-8 

Delta  in  phase  of  problem/issue  detection 

0 

3 

8 

10 

1 

Upfront  cost  of  new  process  (upfront  $) 

0 

3 

-1 

3 

-4 

Per  program  (reoccuring)  cost  (per  program  $) 

0 

10 

5 

9 

0 

Approval  Level  (upfront  risk  and  time) 

0 

7 

1 

-2 

-4 

Coordination  required  to  implement  change  (upfront  time) 

0 

3 

-6 

3 

-7 

Time  to  develop,  train,  implement  the  new  process  (upfront  time) 

0 

-2 

-2 

1 

-7 

Impact  to  program  schedule  (per  program,  time) 

0 

6 

3 

10 

4 

Change  in  documentation  for  programs  (per  program  time  and  $) 

0 

-4 

0 

2 

-10 

System  Sum  (the  entire  group) 

0 

24 

19 

41 

-35 

Average  for  each  Alternative  (the  entire  group) 

0 

1.71 

1.36 

2.93 

-2.50 

System  Sum  (drop  the  highest  lowest  vote) 

0 

28 

21 

42 

-31 

Average  for  each  Alternative  (drop  highest  and  lowest) 

0 

2.33 

1.75 

3.50 

-2.58 

Figure  22.  Pugh  Matrix  Results 


This  figure  summarizes  the  results  of  the  AoA. 


C.  SELECTION  OF  PREFERRED  ALTERNATIVE 

Once  the  Pugh  Matrix  results  were  tabulated,  the  authors  discussed  the  results  based  upon 
the  value  system  discussed  in  Chapter  II.  The  majority  of  the  discussion  focused  on  the  three 
alternatives  with  the  highest  “score”  in  the  Pugh  Matrix:  “Define  and  Apply  Interoperability 
Readiness  Levels  (l/ORLs)”  had  41  pluses,  “Consider  Interoperability  at  all  Milestone  Reviews” 
had  24  pluses  and  “Test  to  Mission  Thread  at  Interoperability  OT”  had  19  pluses.  The  Mission 
Thread  alternative  had  the  most  pluses  in  the  two  measures  of  effectiveness,  but  at  a  much  higher 


67 


anticipated  cost.  Initial  discussions  with  the  project  stakeholders,  specifically  with  an  ASN 
(RDA)  CHSENG  staffer,  had  identified  a  concept  based  on  Mission  Threads  as  being  desirable. 
As  this  alternative  most  reflected  the  original  Project  Proposal,  it  was  a  hard  decision  to  choose 
another  path,  however,  the  authors  discussed  the  Pugh  Results  and  agreed  with  the  analysis  that 
this  alternative  would  be  harder  to  coordinate  between  organizations  and  would  have  a  greater 
negative  impact  on  each  program’s  schedule  and  budget  than  the  other  two  alternatives.  These 
impacts  would  be  due  to  the  change  in  requirements  development  process  up  front  as  well  as  a 
change  in  how  OT  process  and  procedures  are  developed,  carried  out  and  documented.  On  the 
other  hand,  I/ORLs  and  the  evaluation  of  interoperability  at  each  milestone  were  projected  to 
require  less  significant  changes  to  the  current  OT  process. 

The  remaining  two  alternatives  were  viewed  by  the  authors  to  be  similar,  as  it  was  agreed 
that  the  implementation  of  I/ORLs  would  likely  involve  the  evaluation  of  I/ORLs  at  key 
technical  reviews  and  would  therefore  be  inputs  to  the  milestone  decisions.  It  was  also  felt  that 
specifying  that  interoperability  be  assessed  at  each  milestone  was  difficult  without  developing  a 
system  by  which  interoperability  of  a  system  would  be  quantified.  Therefore,  the  authors 
concluded  that  developing  I/ORLs  was  the  best  alternative  and  every  effort  would  also  be  made 
to  meet  the  basic  objectives  of  “Considering  Interoperability  at  each  Milestone  Review”. 
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IV.  DETAIL  PROCESS  DESIGN 


Upon  completion  of  the  Analysis  of  Alternatives,  Detailed  Process  Design  began.  A 
more  detailed  functional  analysis  kicked  off  this  phase  in  which  functions  were  defined.  After 
expanding  on  the  functional  analysis,  synthesis  of  the  process  commenced.  I/ORL  Development 
and  Process  Design  comprised  the  synthesis  portion  of  the  detailed  process  design;  in  an  iterative 
fashion,  I/ORL  Development  and  Process  Design  further  refined  the  system.  Consequently,  a 
more  thorough  discussion  on  activities  in  the  functional  analysis  is  presented  in  this  chapter. 
During  this  phase  it  became  apparent  that  the  preferred  solution  of  "Develop  Interoperability 
Readiness  Levels"  implies  one  activity  only,  yet,  this  process  involves  much  more  than  simply 
developing  LORLs.  A  more  suitable  name  for  this  solution  is  "I/ORL  Assessment  Process" 
which  reflects  the  use  of  Interoperability  Readiness  Levels  as  a  mechanism  to  assess  the  maturity 
of  the  design  of  a  system  with  regard  to  how  well  it  interoperates  with  other  systems  in  support 
of  mission  critical  threads,  and  as  a  means  to  predict  a  system's  ability  to  pass  operational  testing 
with  regard  to  interoperability. 

A.  DETAILED  FUNCTIONAL  ANALYSIS 

To  assess  interoperability  readiness  levels,  it  was  detennined  that  LORLs  would  need  to 
be  evaluated  at  multiple  reviews.  A  more  detailed  approach  to  evaluating  a  program’s  I/ORL 
involves  evaluating  interoperability  at  each  review  leading  up  to  a  milestone  review.  The  top 
level  of  the  detailed  process  to  developing  LORLs  considers  how  this  process  will  be 
implemented  and  includes  the  functions  of: 

•  Evaluate  LORLs  at  selected  reviews 

•  Assess  system  I/ORL  performance 

•  Evaluate  interoperability  results 

•  Take  appropriate  action 

•  Continue  development 

•  Conduct  milestone  reviews 

This  functional  analysis  is  reflected  in  Eigure  23.  These  actions  will  be  discussed  briefly 
in  the  following  sections  with  more  detail  in  Sections  B  and  C. 
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Figure  23.  Functional  Analysis  of  Interoperability  Readiness  Level  Process 

This  figure  summarizes  key  functions  in  the  I/ORL  Process. 


1.  Evaluate  at  Selected  Reviews 


Several  reviews  take  place  during  the  development  of  a  system  as  prescribed  by  the 
SETR  [SETR,  2009].  Interoperability  should  be  considered  at  many  of  these  reviews.  However, 
some  reviews  are  not  technical  and  interoperability  is  not  a  factor  in  these  reviews.  The  authors 
considered  all  of  the  reviews  leading  up  to  milestone  reviews  and  selected  the  reviews  in  which 
interoperability  should  be  considered.  Figure  24  summarizes  the  selected  reviews  leading  up  to 
milestone  reviews. 


Evaluate  at  Selected 
Reviews 


Conduct 

Milestone 

Review 


Evaluate 

/ 

/ 

I 

I 

Interoperability 

l 

I 

Interoperability  j* 


Figure  24.  Selected  Technical  Reviews  for  FORL  Evaluation 

The  Evaluate  at  Selected  Reviews  reflects  a  choice  of  technical  reviews  in  which  I/ORLs  should  he  considered. 
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2.  Assess  System  I/ORL  Performance 

In  order  to  assess  the  system  performance  I/ORL  levels  were  selected  and  described,  and 
a  quantitative  value  was  assigned  to  each  level.  Lower  values  reflect  less  mature  interoperability 
levels  and  increasingly  higher  values  reflect  progressing  maturity  in  a  system's  ability  to  pass 
information  at  interfaces.  This  is  the  scale  against  which  interoperability  will  be  measured; 
interoperability  of  each  interface  of  a  system  must  be  considered  and  assessed. 

3.  Evaluate  Interoperability 

Once  interoperability  assessment  data  are  available,  values  are  evaluated.  I/ORL  values 
are  measured  against  a  threshold  and  objective  level  designated  for  a  particular  review.  The 
values  have  been  assigned  based  on  the  purpose  of  the  review.  An  objective  is  a  quantitative 
level  that  should  be  met.  A  threshold  is  a  minimum  value,  a  level  that  must  be  met. 

4.  Take  Appropriate  Action 

Upon  evaluating  interoperability,  appropriate  action  must  be  taken.  This  action  can  be 
one  of  several  depending  on  the  evaluation.  If  an  I/ORL  value  meets  the  objective,  it  proceeds 
through  the  system  development  process  as  usual.  If  the  I/ORL  obtains  a  value  between  the 
objective  and  threshold  value,  a  system  interoperability  perfonnance  review  must  be  performed. 
If  the  system  fails  to  meet  the  threshold  value  it  must  undergo  a  mini  re-view  or  repeat  the 
review  in  order  to  rectify  interoperability  issues. 

5.  Continue  Development 

The  system  must  proceed  with  development  per  their  tailored  system's  engineering 
process.  The  system  must  continue  to  consider  interoperability  as  it  progresses  with  maturing 
interoperability  readiness. 

6.  Conduct  Milestone  Review 

A  milestone  review  is  conducted  as  usual  with  the  inclusion  of  interoperability 
consideration.  Findings  from  the  LORL  reviews  perfonned  at  technical  reviews  leading  up  to  a 
milestone  review  are  considered,  this  may  include  LORL  evaluations  and  mitigations.  Further, 
concurrence  on  issues  and  LORL  decisions  should  be  obtained  during  the  milestone  review. 
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B.  I/ORL  DEVELOPMENT 

I/ORL  Development  began  with  examining  all  of  the  reviews  outlined  in  the  SETR 
handbook  [SETR,  2009].  Reviews  containing  an  interoperability  component  were  selected.  The 
authors  then  created  an  I/ORL  standard  outlining  I/ORL  levels,  values,  and  descriptions.  Finally 
threshold  and  objective  values  were  assigned  to  relevant  reviews  based  on  outputs  of  each  review 
gleaned  from  the  SETR  handbook. 

1.  Applicable  Reviews 

I/ORLs  are  intended  to  assess  the  state  of  system  interoperability  using  the  systems 
engineering  process  within  the  existing  DoD  framework.  Consequently,  reviews  from  the  SETR 
handbook  were  chosen.  After  inspecting  the  reviews  in  the  SETR  handbook  the  authors  arrived 
at  a  list  of  reviews  that  should  include  an  I/ORL  assessment.  Table  8  reflects  the  selected 
reviews,  their  purpose,  and  a  brief  explanation  on  why  interoperability  should  be  considered  at 
the  chosen  review. 

Table  8.  Technical  Reviews  for  EORL  Assessment 

This  table  shows  the  technical  reviews  in  which  an  I/ORL  assessment  is  applicable. 


Review 

Purpose 

Rationale 

Alternative 

System  Review 
(ASR) 

Reviews  results  of  Material 
Solution  Analysis  phase  and 
assesses  technology  development 
plan  and  preferred  system 
concept. 

This  is  an  early  technical  review, 
interoperability  should  be  considered 
from  the  early  stages  of  development  in 
order  to  successfully  pass  operational 
testing. 

System 
Requirements 
Review  (SRR) 

Assesses  technical  readiness  to 
enter  Engineering  & 

Manufacturing  Development 
Phase. 

This  review  establishes  requirements 
and  assesses  the  developing  system  to 
ensure  a  likely  expectation  that  the 
ultimate  system  will  be  operationally 
effective;  interoperability  must  be 
considered  in  requirements  for 
successful  interoperability. 

System 

Functional 

Review  (SFR) 

Assesses  System  Functional 
Baseline  and  readiness  to  begin 
functional  allocation. 

This  review  is  a  technical  assessment  to 
establish  that  the  system  functional 
baseline  will  likely  be  deemed 
operationally  effective,  interoperability 
must  be  considered  in  functional 
specification  for  success. 

Preliminary 

Design  Review 

Assess  System  Allocated 

Baseline  and  readiness  to  begin 

This  review  is  a  technical  assessment  to 
establish  that  the  physically  allocated 
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Review 

Purpose 

Rationale 

(PDR) 

detailed  design. 

baseline  will  likely  be  deemed 
operational  effective,  interoperability 
must  be  considered  in  the  physical 
architecture  for  successful  passing  of 
infonnation  at  interfaces. 

Software 
Specification 
Review  (SSR) 

Assesses  completeness  of 
software  specification. 

This  review  is  a  technical  assessment  to 
establish  that  the  software  requirements 
baseline  of  the  system  and  its 
preliminary  design  will  likely  be 
deemed  operationally  effective, 
interoperability  must  be  considered  in 
software  for  successful  passing  of  data. 

Critical  Design 
Review  (CDR) 

Assesses  System  Product 

Baseline  and  supports  Design 
Readiness  Review. 

This  review  is  a  technical  assessment  to 
establish  that  the  build  baseline  likely 
be  deemed  operational  effective, 
interoperability  must  be  considered  in 
the  design  in  order  to  successfully  pass 
operational  testing. 

Flight  Readiness 
Review  (FRR) 

Assesses  system  readiness  to 
initiate  and  conduct  flight  tests 
and  flight  operations. 

This  review  is  a  technical  assessment  to 
establish  that  the  configuration  used  in 
flight  testing  will  likely  be  deemed 
operational  effective,  interoperability 
must  be  considered  at  this  review  for 

success. 

Integration 

Readiness 

Review  (IRR) 

Assesses  readiness  of  software 
systems. 

This  review  is  an  assessment  to  ensure 
that  the  hardware  and  software  are 
ready  to  begin  integrated  configuration 
item  testing,  interoperability  must  be 
considered  in  the  configuration  for 
successful  completion  of  operational 
testing. 

Operational  Test 
Readiness 

Review  (OTRR) 

Assesses  system  readiness  to 
proceed  into  Operational  Test 
and  Evaluation  (OT&E). 

Interoperability  maturity  must  be 
considered  to  ensure  that  the  system 
can  proceed  into  OT&E  with  a  high 
probability  that  the  system  will 
successfully  complete  operational 
testing. 

System 
Verification 
Review  (SYR) 

Assesses  system  compliance  with 
functional  baseline. 

This  review  is  an  assessment  to  ensure 
that  the  system  under  review  can 
proceed  into  Low  Rate  Initial 

Production;  interoperability  must  be 
achieved  for  the  system  to  proceed  to 
LRIP. 
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While  most  of  the  reviews  outlined  in  the  SETR  are  applicable  to  interoperability  and 
subsequently  an  I/ORL  assessment,  some  reviews  are  neither  technical  nor  appropriate  for  I/ORL 
consideration.  Table  9  reflects  the  reviews  excluded  from  I/ORL  assessment  with  a  brief 
explanation  as  to  their  rejection. 


Table  9.  Technical  Reviews  Excluded  From  I/ORL  Assessment 

This  table  shows  the  technical  reviews  in  which  an  I/ORL  assessment  is  not  applicable. 


Review 

Purpose 

Rationale 

Initial  Technical 
Review  (ITR) 

Supports  technical  basis  for  initial 
cost  estimates  and  POM  budget 
submissions. 

This  review  focuses  on  cost  and 
budget  versus  technical  issues. 

Integrated 

Baseline  Review 
(IBR) 

Assess  risk  areas  in  contract. 

Produces  Performance  Measurement 
Baseline  to  ensure  technical  scope 
of  work  is  realistically  and 
accurately  scheduled,  has  proper 
resources,  utilizes  correct 
techniques,  and  employs  appropriate 
management  processes. 

This  review  focuses  on  earned 
value  management  versus  technical 
issues. 

Test  Readiness 
Review  (TRR) 

Assesses  system  readiness  to  begin 
Developmental  Test  and  Evaluation 
(DT&E). 

This  review  coincides  closely  with 
the  CDR,  there  will  be  a  desire  to 
test  any  unfavorable  results  before 
addressing  them. 

Production 

Readiness 

Review  (PRR) 

Assesses  system  to  enter  production. 

This  review  is  conducted  after 
Milestone  C  and  operational 
testing. 

Physical 
Configuration 
Audit  (PCA) 

Assesses  the  as-delivered  system  for 
compliance  with  the  produce 
baseline  and  supports  full-rate 
production  decision. 

This  review  is  conducted  after 
Milestone  C  and  operational 
testing. 

In-Service 

Review  (ISR) 

Assesses  the  in-service  technical 
health  of  a  fielded  system  from  a 
risk,  readiness,  and  resources 
perspective. 

This  review  is  conducted  after 
Milestone  C  and  operational 
testing. 

Once  the  appropriate  reviews  were  selected,  the  authors  developed  I/ORL  values  and 
their  meaning. 
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2.  I/ORL  Values 


I/ORLs  are  to  be  used  similarly  to  Technology  Readiness  Levels  (TRLs)  throughout  the 
SETR  process  [SETR,  2009].  After  selecting  reviews  in  which  to  assess  I/ORLs,  meaningful 
I/ORL  values  were  assigned.  It  should  be  noted  that  a  reference  frame  for  IRLs  is  available  in  A 
Systems  Approach  to  Expanding  the  Technology  Readiness  Level  within  Defense  Acquisition 
[Sauser,  2009].  However,  Sauser,  et  al,  inappropriately  applied  mathematical  operation  on  non¬ 
ratio  scale  numbers  in  their  paper.  A  discussion  of  the  issues  with  the  methodology  applied  by 
Sauser,  et  al,  can  be  found  in  The  trouble  with  the  System  Readiness  Level  (SRL)  index  for 
managing  the  acquisition  of  defense  systems  [Kujawski,  2010],  The  authors  wanted  to  ensure  we 
did  not  duplicate  that  error.  Therefore,  the  authors  decided  to  develop  our  own  I/ORL  level 
designations  to  tailor  them  to  our  process.  In  order  to  keep  the  process  simple  and  consequently 
practicable,  levels  of  1  to  9  were  chosen.  A  value  of  1  is  considered  the  least  mature  as  it 
pertains  to  interoperability,  and  a  value  of  9  is  the  most  mature  as  it  pertains  to  interoperability. 

It  is  important  to  note  that  the  authors  are  choosing  to  describe  the  acronym  I/ORL  as 
Interoperability  Readiness  Level  versus  IRL  as  Integration  Readiness  Level  example  presented 
in  the  AoA  section.  The  authors  chose  to  use  “interoperability”  as  it  aligns  more  closely  with  the 
issues  in  this  project.  Further,  the  authors  are  expanding  upon  the  example  presented  in  the  AoA 
and  are  tailoring  I/ORLs  to  the  issue  of  systems  failing  operational  testing.  As  a  result,  the 
interoperability  readiness  levels  appointed  by  the  authors  are  outlined  below: 

I/ORL  1 

1)  The  physical  interfaces  and  software  scope  of  the  preferred  system  solution  are 
sufficiently  understood  to  begin  development  with  low  technical  risk. 

2)  The  Program  has  determined  which  legacy  and  other  development  systems  the  new 
system  must  interface  with. 

Any  other  systems  that  interface  with  the  developing  system  must  be  identified.  This 
includes  systems  that  physically  connect  to  the  new  system  as  well  as  software  and  data  sharing 
communications.  The  details  of  the  connections  do  not  have  to  be  explicit.  This  I/ORL  simply 
states  that  a  connection  will  be  needed.  Completing  this  I/ORL  will  aid  in  the  development  of  a 
full  list  of  mission  critical  interoperability  requirements  which  will  be  evaluated  individually 
during  successive  reviews. 
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I/ORL  2 

1)  A  design/technology  maturity  risk  analysis  has  been  completed.  Interoperability  issues 
that  could  impact  cost,  schedule  or  system  performance  have  been  identified. 

2)  An  AO  A  is  complete  and  addresses  interoperability  between  the  new  system ’s  interfaces 
and  other  applicable  systems  that  require  interoperability  with  the  new  system. 

An  assessment  of  current  technology  is  complete;  the  risk  analysis  provides  the  Program 
Manager  with  valuable  insight  into  whether  the  new  system  will  be  fully  interoperable  with 
existing  systems.  Schedule  and  funding  changes  can  be  made  early  to  address  any  high  risk 
system  interfaces.  The  AOA  is  used  to  determine  if  the  various  interfaces  are  being  used 
optimally;  there  may  be  alternative  configurations  that  are  succinct  then  the  originally  planned 
new  System  of  Systems.  The  AOA  can  further  reduce  risk  by  ensuring  the  optimal  system 
interoperation  configuration  is  chosen  and  may  identify  unnecessary  redundancy,  reduce  the 
quantity  of  interfaces  or  maximize  the  efficiency  of  the  interfaces  that  interoperate. 

I/ORL  3 

1)  The  hardware  interface  specification  is  sufficiently  detailed  to  include  requirements  such 
as  MIL-STD,  ISO,  IEEE,  etc. 

2)  The  software  interface  specification  is  sufficiently  detailed  to  include  requirements  such 
as  programming  languages,  protocols,  and  standards. 

I/ORL  3  does  not  evaluate  the  new  system’s  interface  but  rather  focuses  on  the 
documentation  of  the  interface  requirements  and  design  generation.  All  of  the  connections, 
physical  and  electronic  must  have  documented  standards.  A  large  portion  of  this  I/ORL  is 
completing  sufficient  research  on  the  other  systems  that  the  new  system’s  interfaces  will 
interface  with  in  order  to  detennine  what  standards  those  systems  already  use/require  and  ensure 
that  the  standards  required  for  the  new  system’s  interfaces  are  backwards  compatible.  Further, 
any  upgrades  required  to  legacy  systems  that  are  based  on  obsolete  standards  should  be 
identified  at  this  point,  for  early  development. 

I/ORL  4 

1)  The  interoperability  of  the  interface  shall  be  demonstrated  using  its  respective 
standards. 

I/ORL  4  represents  a  transition  from  a  conceptual  interface  to  one  that  has  been  realized 
and  is  a  combination  of  research  and  testing.  The  objective  of  this  evaluation  is  to  demonstrate 
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that  the  interface  is  mature.  It  is  not  intended  that  development  of  any  portion  of  the  system  be 
finished  at  this  point.  This  is  just  a  technology  demonstration  or  documentation  of  technology 
demonstration  performed  elsewhere  using  this  technology  to  demonstrate  that  interoperability  is 
achievable  without  regard  to  the  test  conditions,  thus  this  can  be  done  in  the  lab  or  in  the  field. 

I/ORL  5 

1)  Interface  design  is  complete  and  has  undergone  an  independent  review  for 
interoperability  deficiencies. 

As  a  system  approaches  its  product  baseline  each  interface  must  be  complete  and  have 
undergone  a  review  for  interoperability  deficiencies.  As  a  prerequisite  of  this  review  the 
interface  shall  have  been  reviewed  by  an  independent  party,  not  the  SE  team  or  contractor 
developing  the  new  system,  to  ensure  interoperability  has  been  addressed.  This  evaluation  is 
critical  to  ensuring  that  time  and  money  is  not  wasted  on  building  a  design  that  is  incomplete  or 
unachievable. 

I/ORL  6 

1)  Interface  test  article  components  have  successfully  completed  Developmental  Test 
and  Evaluation  for  interoperability. 

The  individual  components  have  been  developed  and  have  successfully  completed  DT&E 
(bench  or  lab  testing);  DT&E  must  include  evaluation  of  the  interoperability  requirements  of  the 
interfaces  being  tested.  The  interfaces  are  ready  for  integration  into  larger  sub-systems  and 
eventually  into  the  full  system. 

I/ORL  7 

1)  All  interfaces  have  been  tested  between  the  new  system  and  the  legacy  system(s)  it 

interacts  with. 

2)  Interface  interoperability  with  all  other  systems  has  been  verified  through  simulated 

operational  scenarios. 

I/ORL  7  must  be  accomplished  prior  to  entering  OT&E.  Modeling  and  Simulation  may 
be  used  to  verify  that  interoperability  of  the  entire  system  should  work  during  actual  testing. 
Furthennore,  each  interface  has  been  verified  through  interoperability  testing  to  ensure  it 
functions  properly  with  other  system’s  components,  such  as  completing  fit  testing  for  physical 
interconnections,  completing  radio  communication  checks  for  systems  with  RF  or  wireless 
communications,  and  connecting  computers  in  the  new  system  to  those  in  legacy  systems  to 
ensure  data  transfer  is  correctly  established,  etc. 
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I/ORL  7F  (only  for  system  requiring  flight  based  testing) 

1)  Interoperability  with  ail  other  interface  has  been  verified  through  ground  based 
verification  testing. 

I/ORL  7F  is  an  additional  requirement  for  I/ORL  7  for  any  system  that  will  undergo 
flight  based  testing,  either  as  an  aircraft,  new  flight  system,  rocket,  weapons  system  on  an 
aircraft,  etc.  I/ORL  7F  is  intended  to  eliminate  interoperability  failures  during  flight  testing, 
which  could  potentially  lead  to  loss  of  the  test  system,  air  frame  or  human  life  in  the  event  of  a 
failure  and  crash.  Completing  I/ORL  7F  is  accomplished  by  simulating  the  entire  flight  test  on 
the  ground.  It  ensures  that  all  interfaces  and  sub-systems  of  the  new  system  are  functioning 
properly  with  each  other  as  well  as  other  systems.  It  ensures  that  all  communications  lines  with 
other  systems  are  functional.  While  numerous  ground  tests  are  currently  performed  prior  to 
flight  testing,  achieving  I/ORL7F  ensures  that  interoperability  is  addressed  and  something  is  not 
overlooked  before  the  system  is  actually  engaged  in  flight  operations  off  of  the  ground. 

I/ORL  8 

1)  Ail  interoperability  issues  discovered  during  OT&E  have  been  mitigated  (or  none 
were  present).  The  system  and  its  interfaces  are  now  mission  qualified. 

I/ORL  8  is  achieved  by  either  having  no  interoperability  issues  during  OT&E  or  by 
documenting  the  steps  taken  to  mitigate  any  interoperability  deficiencies.  Interoperability  issues 
that  are  below  requirements,  but  do  not  cause  system  failure;  such  as  slower  than  required 
(desired)  data  exchange  rates,  or  physical  connections  outside  of  expected  tolerance  must  either 
be  corrected  or  the  program  manager  may  accept  the  deficiency  and  reduced  capability  from  the 
initial  requirements.  LORL  8  is  an  important  milestone  for  the  system  as  I/ORL  8  indicates  that 
there  are  no  remaining  interoperability  issues  that  have  not  been  corrected  or  accepted  and  the 
system  is  ready  for  use  on  the  battlefield. 

I/ORL  9 

1)  The  system  has  been  proven  to  be  interoperable  during  operational  use  (Mission 
Proven) 

LORL  9  is  achieved  once  a  system  is  deployed  and  has  demonstrated  successful  mission 
completion  (either  actual  or  a  training  mission)  in  the  fleet/field.  Any  deficiencies  discovered  in 
the  field/fleet  previously  not  experienced  during  other  testing  have  been  mitigated  or  accepted 
(just  as  in  LORL  8).  If  OT&E  is  completed  in  the  field/fleet  and  not  at  a  test  command  or  test 
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range  then  the  system  may  achieve  I/ORL  9  simultaneously  with  I/ORL  8  once  any 
interoperability  issues  discovered  have  been  mitigated.  An  I/ORL  is  intended  to  signify  that  the 
system  and  all  its  interfaces  have  been  proven  to  be  interoperable. 

3.  I/ORL  Values  for  Reviews 

After  selecting  reviews  in  which  to  assess  I/ORLs,  and  those  values  were  developed, 
I/ORL  levels  were  assigned  to  applicable  reviews.  For  each  applicable  review  an  objective 
I/ORL  value  and  a  threshold  value  was  assigned.  The  objective  value  is  the  level  that  should  be 
met  at  a  particular  review.  A  threshold  value  is  the  minimum  value  that  the  interface  must  meet 
at  the  review.  These  values  are  based  on  the  outcome  of  the  review  as  described  in  the  SETR 
handbook.  If  this  value  is  not  met,  the  system  cannot  proceed  without  further  action.  The 
process  or  appropriate  action  to  take  after  failing  to  meet  a  threshold  value  is  described  in  the 
Process  Design  section. 

Some  reviews  have  a  hard  minimum  LORL  requirement  or  threshold  that  must  be  met 
with  no  objective  beyond  that.  Consequently  these  reviews  have  the  same  threshold  and 
objective  value  in  the  table.  The  applicable  reviews,  exit  criteria  as  they  pertain  to 
interoperability,  objective,  and  threshold  values  are  outlined  in  Table  10. 

Table  10.  LORL  Values  for  Each  Review 
This  table  shows  the  I/ORL  levels  and  values  needed  at  each  review’  [SETR,  2009] . 


Review 

Exit  Criteria 

Threshold 

Objective 

ASR 

-Is/ Are  the  preferred  system  solutions(s)  sufficiently 
detailed  and  understood  to  enable  entry  into  Technology 
Development  with  low  technical  risk? 

-Is  the  system  software  scope  and  complexity 
sufficiently  understood  and  addressed  in  the  Technology 
Development  plan  to  enable  low  software  technical  risk? 
-Are  the  risks  known  and  manageable  for  Technology 
Development? 

1 

2 

SRR 

-Are  the  system  requirements  sufficiently  detailed  and 
understood  to  enable  system  functional  definition  and 
functional  decomposition? 

-Is  the  architecture  adequately  structured  to  support  both 
explicit  and  implied  system  attributes? 

-Are  the  risks  known  and  manageable  for  design  and 
development? 

-Are  the  Family  of  System/System  of  Systems 

2 

3 
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Review 

Exit  Criteria 

Threshold 

Objective 

(FoS/SoS)  requirements  properly  allocated  and 
approved? 

-Did  the  Technology  Development  phase  sufficiently 
reduce  development  risk? 

SFR 

-Are  the  system  functional  requirements  sufficiently 
detailed  and  understood  to  enable  system  design  to 
proceed? 

-Are  adequate  processes  and  metrics  in  place  for  the 
program  to  succeed? 

-Are  the  risks  known  and  manageable  for  design  and 
development? 

3 

3 

PDR 

-Does  the  status  of  the  technical  effort  and  design 
indicate  OPEVAL  success  (operationally  suitable  and 
effective)? 

-Has  the  system  allocated  baseline  been  established  and 
documented  to  enable  detailed  design  to  proceed  with 
proper  configuration  management? 

-Are  adequate  processes  and  metrics  in  place  for  the 
program  to  succeed? 

-Are  the  risks  known  and  manageable  for  DT/OT? 

3 

4 

SSR 

-Inputs,  processing,  and  outputs  are  defined  and 
accepted  for  all  functions 

-All  interfaces  between  the  software  configuration  item 
and  all  other  configuration  items  both  internal  and 
external  to  the  system  are  defined  and  accepted  as  stable. 
In  particular,  interoperability  requirements  are  fully 
identified  and  defined,  accepted,  and  correlated  to 
mission  requirements  and  scenarios. 

-All  interface-level  data  elements  are  defined  and 
accepted,  including  data  type,  size,  fonnat,  units,  range 
of  values,  accuracy  and  precision 
-SW  development  processes  are  fully  defined  in  the  SDP 
or  equivalent  document  (e.g.,  Software  Standards  and 
Procedures  Manual  (SSPM)),  and  are  accepted  as 
appropriate  for  coding  and  unit  test. 

-Risks  are  identified  in  a  Risk  Database  and  have 
mitigation  plans  in  place  that  are  compatible  with  the 

SW  development  schedule. 

4 

4 

CDR 

-Does  the  status  of  the  technical  effort  and  design 
indicate  OPEVAL  success  (operationally  suitable  and 
effective)? 

-Has  the  system  product  baseline  been  established  and 
documented  to  enable  hardware  fabrication  and  software 
coding  to  proceed  with  proper  configuration 

5 

5 
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Review 

Exit  Criteria 

Threshold 

Objective 

management? 

-Are  adequate  processes  and  metrics  in  place  for  the 
program  to  succeed? 

-Are  the  risks  known  and  manageable? 

IRR 

-  The  IRR  is  considered  complete  when  all  draft  RFAs 
are  signed  off,  and  an  acceptable  level  of  program  risk  is 
ascertained. 

6 

6 

FRR 

-The  FRR  is  considered  complete  when  all  draft  RFAs 
are  signed  off,  and  an  acceptable  level  of  program  risk  is 
ascertained. 

7&7F 

7&7F 

OTRR 

-The  OTRR  is  considered  complete  when  all 
requirements  for  Navy  Certification  of  Readiness  for  OT 
are  complete. 

-For  programs  employing  software,  there  are  no 
unresolved  priority  1  or  2  software  problem  reports 
(SPR),  and  all  priority  3  problems  are  documented  with 
appropriate  impact  analyses. 

7 

7 

SYR 

-Does  the  status  of  the  technical  effort  and  system 
indicate  operational  test  success  (operationally  suitable 
and  effective)? 

-Are  adequate  processes  and  metrics  in  place  for  the 
program  to  succeed? 

-Are  the  risks  known  and  manageable? 

-Are  the  system  requirements  understood  to  the  level 
appropriate  for  this  review? 

8 

8 

C.  I/ORL  PROCESS  DESIGN 

Once  the  authors  had  generated  the  definition  of  the  I/ORLs  at  each  level,  the  process  for 
assigning  and  evaluating  I/ORLs  was  generated.  In  developing  this  process,  the  authors 
researched  the  history  of  interoperability  measurement  and  keyed  in  on  the  most  widely  used 
interoperability  measurement  systems.  The  most  notable  one  was  the  i-Score  methodology 
[Ford,  2007;  Ford,  2008]:  an  interoperability  measurement  technique  for  generating  a  system- 
wide  interoperability  value  based  on  a  numerical  analysis  of  mission  threads.  While  i-Score 
approaches  interoperability  from  a  different  perspective,  the  authors  leveraged  certain  core 
concepts  in  the  I/ORL  system. 
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The  overall  process  that  was  generated  was  split  into  two  major  phases:  work  done  before 
the  milestone  review  process  and  work  done  at  each  of  the  milestone  reviews.  The  steps 
included  in  each  of  the  phases  are  shown  in  Figures  25  and  26. 


Mission  Critical 
Thread  Analysis 


Figure  25. 1/ORL  Process  Steps  -  Pre -Review 

These  are  the  key  steps  that  occur  prior  to  the  Milestone  Review’. 


Identify  CSIs 


Figure  26. 1/ORL  Process  Steps  -  Each  Review 

These  are  the  key  steps  that  occur  at  each  Technical  Review. 


Detailed  descriptions  of  the  steps  involved  with  assigning  and  evaluating  a  system  for 
I/ORLs  are  listed  in  the  sections  below. 

1.  Identify  and  Analyze  Mission  Critical  Threads 

Prior  to  conducting  the  analysis  of  Interoperability  Readiness  Levels  (EORLs),  it  is 
important  to  understand  the  critical  interfaces  of  a  system.  To  outline  those  critical  interfaces, 
one  must  first  analyze  the  critical  mission  threads  that  are  involved  with  accomplishing  the 
mission. 

The  team’s  concept  of  initializing  the  necessary  mission  thread  and  defining  the 
necessary  interfaces  are  shared  with  i-Score’s  methodology,  [Ford,  2007].  The  i-Score 
methodology  diagrams  an  operational  thread,  and  from  this  diagram  identifies  the  systems  that 
require  interoperability.  These  systems  are  then  numerically  assessed  for  interoperability  and 
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recorded  in  a  matrix,  which  is  used  to  generate  a  score.  The  team’s  method  varies  from  the  later 
part  of  their  process,  but  share  a  similar  methodology  for  identifying  interfaces. 

Before  any  mission  threads  are  analyzed,  a  review  team  consisting  of  key  stakeholders 
and  program  managers  must  identify  the  critical  mission  threads  out  of  the  many  mission  threads 
that  may  exist  for  the  system.  This  narrows  down  the  scope  of  the  analysis  to  something  more 
manageable  while  still  maintaining  a  high  degree  of  fidelity.  The  review  team  should  use  the 
following  or  similar  definition  of  mission  criticality  as  their  guide  for  selecting  the  appropriate 
threads  for  analysis.  The  following  definition  is  provided  in  Joint  Chiefs  of  Staff  Instruction 
(CJCSI)  6212.01E: 

An  operational  and  technical  description  of  the  end  to  end  set  of  activities 
and  systems  that  accomplish  the  execution  of  a  joint  mission  [CJCSI 
6212. 01E,  2008]. 

The  review  team  for  the  mission  and  the  system  involved  must  make  a  joint 
detennination  that  the  thread  in  question  meets  the  above  definition. 

Similar  to  the  i-Score  methodology’s  Step  1  [Ford,  2007]  of  diagramming  the  operational 
thread  and  defining  the  set  of  supporting  systems,  the  mission  threads  involved  with  the  system 
are  evaluated  for  mission  criticality  then  diagramed  for  analysis.  This  involves  all  of  the  various 
steps  in  the  functional  flow  of  the  system  to  accomplish  that  mission  thread.  Once  completed, 
the  functional  flow  elements  must  be  traced  back  to  their  individual  system  components  and  the 
interfaces  between  components  must  be  identified.  This  method  is  different  from  the  i-Score 
methodology  in  that  our  process  tracks  the  interfaces  between  systems,  not  the  actual  systems 
themselves.  The  interfaces  outlined  here  are  the  interfaces  associated  with  the  mission  thread. 

The  interfaces  between  mission  functions,  components,  and  organizations  are  then  cross- 
referenced  with  the  mission  thread  to  determine  their  criticality  towards  the  mission  thread.  If  an 
interface  is  not  deemed  critical,  then  that  interface  will  not  be  considered  at  this  level  of  review. 
The  interfaces  that  are  deemed  critical  will  then  be  identified  as  Critical  System  Interfaces 
(CSIs).  These  interfaces  will  be  then  documented  as  the  interfaces  at  which  the  I/ORLs  of  the 
system  will  be  evaluated  during  each  review.  All  other  interfaces  are  recognized  and  should  be 
scrutinized  prior  to  each  milestone  review,  but  the  I/ORL  detennination  will  only  be  made  on  the 
CSIs  to  focus  the  attention  on  the  aspects  that  are  important  to  mission  success. 
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2.  Identify  Required  Personnel  /  Research 

After  the  Mission  Critical  Threads  and  their  respective  CSIs  are  identified,  the  respective 
Subject  Matter  Experts  (SME)  /  organization  representatives  for  each  of  the  critical  system 
interfaces  meet  with  an  unbiased  third  party  (either  government  or  contractor)  to  assess  the 
validity  of  the  mission  thread  and  interfaces.  This  team  of  SMEs  and  3rd  party  representatives 
will  eventually  assign  the  I/ORLs  prior  to  the  System  Engineering  Technical  Review.  Each 
SME  or  organization  representative  must  review  the  mission  thread  and  interfaces,  and  be  ready 
to  discuss  each  aspect  of  the  interface. 

3.  Review  Mission  Threads  and  CSIs 

As  the  systems  move  through  the  acquisition  cycle,  the  critical  threads  may  change  as 
well  as  the  various  system  interfaces  that  are  involved.  This  means  that  the  above  analysis  for 
detennining  the  CSIs  should  be  revised  prior  to  each  EORL  determination  in  support  of 
milestone  reviews.  It  is  not  necessary  to  start  the  mission  thread  analysis  from  scratch,  but 
verifying  that  all  of  the  analysis  is  still  applicable  in  light  of  any  major  system  changes  should  be 
completed.  Each  SME/  organization  representative  must  discuss  the  interfaces  in  order  to 
approve,  remove,  or  add  to  the  given  critical  system  interfaces. 

4.  Discuss  Interfaces 

Each  of  the  CSIs  must  then  be  analyzed  for  interoperability  problems.  While  there  are 
many  different  ways  to  accomplish  that,  a  list  of  suggested  evaluation  criteria  is  described  in 
Table  11. 
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Table  11.  Suggested  Evaluation  Criteria 

This  table  describes  the  suggested  evaluation  criteria  for  I/ORLs 


Topic 

Criteria 

Physical  Interfaces 

Ensure  physical  connections  are  adequate. 

Data  Type 

Ensure  data  from  one  system/organization  is  compatible  with 
the  other  system/organization. 

Data  Amount 

Ensures  that  the  amount  of  data  transferred  can  be 
utilized/stored  properly  within  the  other  systems. 

Bandwidth 

Ensure  systems  can  accommodate  the  rate  at  which  data  is 
received. 

Hardware/Software 

Requirements 

Ensure  hardware  and  software  is  compatible. 

Data  Security 

Ensure  data  is  correctly  secured. 

Human  System  Interface 

Ensure  usability  with  software  /  hardware. 

Physical  interfaces  would  be  discussed  during  the  meeting  to  ensure  the  connections 
between  the  systems  are  correctly  designed  and  compatible.  This  is  to  ensure  that  the  interface  is 
adequate  to  meet  the  requirements  of  data  transfer. 

The  type  of  data  is  an  important  factor  to  consider  during  these  discussions  to  ensure  that 
the  data  transferred  from  one  system  is  compatible  with  another  system.  SMEs  will  provide  data 
requirements  for  their  system  and  ensure  that  the  system  will  receive  all  the  information  required 
to  complete  its  own  task.  It  is  important  to  consider  the  quality  of  data  type  that  is  transferred 
and  if  the  data  type  will  always  be  readily  available.  Data  translators  can  be  discussed  if 
required. 

The  amount  of  data  must  be  considered  by  each  system.  Systems  must  account  for  the 
amount  of  data  being  transferred  to  ensure  that  the  systems  are  capable  of  storing  and  utilizing 
the  amount  of  data  received. 

Bandwidth  must  be  considered  to  ensure  each  system  can  accept  the  rate  at  which  data 
flows.  The  systems  must  consider  the  rate  at  which  data  is  transferred  at,  and  the  rate  at  which 
data  is  received. 

Hardware  and  software  are  often  built  as  separate  elements.  This  topic  must  be  discussed 
to  ensure  the  hardware  meets  the  required  specifications  to  adequately  run  the  software.  As 
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software  development  proceeds,  requirement  changes  forces  alteration  of  the  code.  This  step 
will  ensure  hardware  adapts  to  these  software  alterations. 

Data  security  must  be  considered  during  these  meetings  as  well.  The  data  be  within  the 
correct  classification,  and  ensure  that  the  data  is  not  leaked  into  an  improper  classification. 

Human  System  Interfaces  is  important  when  considering  how  different  organizations 
interface  with  software  or  hardware.  These  interfaces  must  be  considered  in  order  to  create, 
perfonn,  and  support  the  mission  thread. 

Any  issues  must  be  recorded  and  discussed  during  the  risk  mitigation  steps.  Mitigations 
steps  for  these  issues  will  be  created  after  identifying  the  appropriate  I/ORL  levels  for  the 
system.  At  this  point,  each  of  the  CSIs  will  have  an  individual  I/ORL  value  assigned  for 
evaluation  at  the  associated  milestone  review. 

5.  Compare  TRL/I/ORL  Values  to  Prescribed  Threshold  and  Objective  Values 

As  a  system  progresses  through  its  various  reviews,  a  program  that  is  on  track  should 
progress  in  a  predicable  manner.  Given  assigned  threshold  and  objective  I/ORL  values  for  each 
review,  each  interface  of  the  system  can  be  evaluated  to  see  if  it  is  on  or  behind  schedule  for  its 
I/ORL  requirements. 

Should  a  system  or  any  of  its  components  fail  to  meet  the  required  threshold  at  a  given 
review,  it  will  need  repeat  the  review  or  conduct  a  mini  re-review.  The  decision  to  repeat  the 
review  or  conduct  a  mini  re-review  will  be  up  to  the  individual  program;  however  it  should 
consider  the  degree  to  which  the  system  failed  to  meet  the  thresholds. 

In  the  event  that  the  LORL  review  yields  results  between  the  objective  and  threshold 
values  the  program  manager  will  need  to  demonstrate  a  risk  mitigation  plan.  The  purpose  of  this 
is  to  address  potential  trouble  areas  in  more  detail  before  they  delay  the  system.  This  approach 
highlights  potential  system  interoperability  issues  allowing  them  to  be  resolved  earlier  in  the 
system  development. 

Components  or  systems  that  exceed  the  objective  LORL  requirement  for  a  given  review 
do  not  require  further  action.  These  systems  or  components  will  be  evaluated  at  the  next  review. 
Ultimately  it  is  desirable  that  all  components  in  a  system  meet  or  exceed  the  objective 
throughout  development.  This  decision  criteria  is  summarized  in  Figure  27  below. 
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The  system  is  evaluated 
against  two  numbers: 
objective  and  threshold 


Milestone  Pass 


Above  the  objective:  system 
passes  without  further 
analysis 


Between  the  objective  and 
threshold:  system  passes 
provided  sufficient  justification 
of  risk 


< -  I RL  Objective 


Milestone  Pass, 
with  Justification 

< -  IRLThreshold 


Below  the  threshold:  system 
cannot  move  past  the 
milestone  review 


Milestone  Fail 


Figure  27.  Milestone  Evaluation  Criteria 

These  are  the  decision  criteria  for  I/ORLs  at  Each  Milestone. 


6.  Perform  risk  mitigation  on  interfaces  that  do  not  meet  objective  values 

For  interfaces  that  fail  to  meet  the  objective  values  for  the  assigned  review  a  system 
interoperability  performance  review  will  need  to  be  perfonned.  The  objective  of  this  review  is  to 
develop  a  mitigation  plan  to  improve  the  I/ORL  levels  of  interfaces  that  failed  to  meet  the 
objective  I/ORL  values.  It  also  provides  a  means  to  track  the  cost  and  schedule  impact  of 
interfaces  that  failed  to  meet  the  objective  I/ORL  values.  This  is  shown  in  Table  12. 
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Table  12.  System  Interoperability  Performance  Review 

This  table  describes  two  examples  of  the  I/ORL  evaluation  process  A3  shows  an  example  of  an  interface  that  is 
lower  than  the  objective  and  greater  than  the  threshold.  To  pass,  it  must  be  at  the  objective  or  above  the  threshold 

with  a  mitigation  step. 


System  or 

component 

interface 

Description 

I/ORL 

value 

I/ORL 

threshold 

I/ORL 

objective 

Mitigation 

Cost 

Schedule 

Antidpated 
LORL  value 

Pass 

At 

Example  1 

4 

2 

4 

Yes 

A3 

Example  2 

3 

2 

4 

Establish 

common 

standards 

$20K 

3  weeks 

4 

Yes 

B7 

Example  3 

1 

2 

4 

Define  interface, 

ID  com¬ 
munication 
requirements 

$60K 

5  weeks 

4 

No 

In  the  first  block  the  system  or  component  interface  is  identified.  This  identification  is 
intended  to  be  a  shorthand  notation  that  is  used  to  track  a  particular  interface.  The  second  block 
provides  a  description  of  the  interface.  This  is  intended  to  augment  the  infonnation  in  the  first 
block  and  provide  sufficient  information  to  understand  the  function  of  the  interface.  The  third 
block  is  the  current  TORL  value  for  the  given  interface.  The  fourth  block,  TORL  threshold, 
denotes  the  minimum  required  TORL  value  at  the  given  review  and  is  intended  as  a  reference 
value.  An  interface  can  still  pass  the  review  if  it  is  above  the  threshold  value;  however,  it  must 
have  a  mitigation  plan  approved  by  the  SETR  review  board  in  order  to  pass.  The  fifth  box  is  the 
objective  TORL  value  for  the  given  review.  Together,  boxes  three  through  five  provide  a  picture 
of  how  well  the  system’s  interface  has  been  developed.  The  sixth  box  is  intended  to  provide  a 
mitigation  strategy.  It  will  be  the  responsibility  of  the  PM  to  develop  a  strategy  to  improve  the 
TORL  of  this  interface.  The  objective  of  the  mitigation  plan  is  to  outline  how  the  given  interface 
will  improve  its  interoperability.  The  seventh  box  is  intended  to  capture  the  costs  associated 
with  the  mitigation  plan.  This  will  be  used  to  track  the  cost  of  improving  interoperability  and 
may  be  useful  for  justifying  changes  to  the  budget.  The  eighth  box  captures  the  impact  to  the 
schedule  as  a  result  of  implementing  the  proposed  mitigation.  Boxes  seven  and  eight  measure 
the  impact  of  considering  interoperability  throughout  the  design  and  development  process,  while 
it  may  appear  to  that  this  process  has  the  potential  to  add  significant  costs  and  delays,  one  must 
consider  the  impact  of  not  addressing  these  issues  early.  The  ninth  box  provides  an  anticipated 
TORL  value,  this  is  the  TORL  value  that  the  PM  anticipates  achieving  after  completing  the 
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mitigation  described  in  box  seven.  The  last  box,  box  ten,  indicates  if  whether  the  given  interface 
passed  the  review.  An  interface  will  pass  a  review  provided  the  TORL  value  (box  3)  is  greater 
than  the  threshold  TORL  value  (box  4)  and  a  mitigation  plan  (box  6)  with  a  favorable  anticipated 
TORL  value  (box  9)  are  provided.  Should  an  interface  fail  to  meet  this,  it  will  be  marked  as 
failing,  meaning  that  further  development  is  required  prior  to  the  system  passing  the  review. 

As  discussed  previously,  components  or  systems  with  interoperability  failures  will  need 
to  repeat  the  review  or  conduct  a  mini  review  of  the  interoperability  failures.  The  decision  to 
repeat  the  review  or  conduct  a  follow-up  review  will  be  up  to  the  individual  program;  however  it 
should  consider  the  degree  to  which  the  system  failed  to  meet  the  thresholds  and  the  risk 
associated  with  the  interface. 
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V. 


PROCESS  VALIDATION 


To  determine  the  effectiveness  of  the  I/ORL  design,  the  authors  developed  a  two-pronged 
methodology.  The  first  method  used  was  a  tabletop  exercise  designed  to  demonstrate  the 
functional  details  of  the  I/ORL  Process  as  it  tracks  the  interoperability  of  a  fictitious  system. 
Although  a  tabletop  exercise  will  not  show  general  applicability  to  all  systems,  it  will 
demonstrate  the  functionality  of  the  approach  presented  in  this  paper.  The  second  method  is  a 
mathematical  simulation  designed  to  track  the  effectiveness  of  the  I/ORL  Process  if  applied 
across  the  Navy  and  DoD.  The  application  of  I/ORLs  and  the  progression  of  the  system  as  it 
moves  through  the  SE  process  is  abstracted,  but  overall  effectiveness  can  be  detennined. 

A.  TABLE  TOP  EXERCISE 

The  authors  simulated  the  LORL  Process  on  a  system  using  a  Table  Top  Exercise  (TTX), 
illustrating  how  the  LORL  Process  could  be  perfonned  at  System  Engineering  Technical 
Reviews.  This  validated  the  process  and  improved  on  gaps  within  the  process.  The  TTX 
included  real-world  scenarios  and  role-plays  that  demonstrated  knowledge  of  policies  and 
procedures  of  the  LORL.  The  TTX  consisted  of  developing  scripts  to  guide  the  exercises, 
separating  the  team  into  role  players,  and  executing  the  LORL  Process.  Further,  the  team 
identified  shortcomings  in  the  LORL  Process,  created  solutions,  and  applied  improvements  to  a 
following  TTX. 

1.  Development  of  the  Table  Top  Exercise 

A  TTX  is  a  discussion-based  walkthrough  of  a  proposed  process  [Radow,  2007].  It  is  a 
method  used  to  review  and  test  a  plan,  allowing  for  interjection  of  unexpected  scenarios  or 
events  in  order  to  vet  many  aspects  of  the  proposed  process.  A  TTX  facilitates  process 
improvement  and  training. 

a.  Selection  of  Reviews 

As  Chapter  IV  of  this  report  describes,  the  LORL  Process  is  to  be  performed  at  1 1 
reviews,  which  are  outlined  in  the  SETR  [SETR,  2009].  Ideally  the  new  LORL  Process  would 
be  validated  via  a  TTX  on  all  of  these  reviews;  however,  the  team  selected  a  subset  of  reviews 
due  to  time  constraints:  ASR  and  OTRR.  These  reviews  are  in  different  phases  of  maturity  for  a 
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system  under  development  and  allowed  for  “lessons  learned”  to  be  implemented  in  at  least  one 
subsequent  review. 


b.  Selection  of  a  Program 

The  authors  investigated  actual  DoD  systems  and  fictional  systems  on  which  to  exercise 
the  I/ORL  Process.  By  choosing  an  actual  system,  it  would  allow  the  team  to  compare  the  data 
from  the  TTX  to  real  data,  showing  how  the  process  can  actually  improve  a  system’s  chances  of 
passing  OT.  Upon  investigating  actual  DoD  systems,  the  authors  discovered  that  data  was  not 
readily  available,  in  part  due  to  the  classification  of  data.  As  a  result,  the  authors  turned  to 
hypothetical  systems  to  use  for  the  TTX.  Due  to  a  theoretical  system’s  flexibility,  it  gave  the 
authors  the  ability  generate  and  find  data  more  readily  and  constrain  a  system’s  complexity; 
however,  it  meant  the  loss  of  the  ability  to  compare  the  system  to  real  data.  A  list  of  pros  and 
cons  in  comparing  an  actual  system  to  a  theoretical  system  is  presented  in  Table  13. 


Table  13.  Comparison  of  Actual  versus  Theoretical  System  for  TTX 

This  table  describes  the  benefits  and  shortfalls  of  using  a  theoretical  system  rather  than  an  actual  system  for  the 

Table  Top  exercise. 


Actual  System 

Theoretical  System 

Pros 

Cons 

Pros 

Cons 

Comparable  to 
real  data 

Requires  extensive 
knowledge  on  a  system 

Infonnation  readily 
available 

Unable  to  verify  data 
in  an  actual  system 

Not  all  infonnation  is 
readily  available 

System  can  be 
constrained 

Information  may  be 
classified 

Faster  to  obtain  data 

More  flexibility  in 
finding  data 

Based  on  the  pros  and  cons,  the  authors  decided  to  use  a  theoretical  system.  The  system 
used  was  the  Advanced  Combat  Integrated  Helmet  (ACIH)  [ACIH,  Garcia  et  al.],  which  is  a 
helmet  with  integrated  communications,  night  vision,  thermal  vision,  navigation,  and  mapping 
tools.  It  includes  many  internal  and  external  interfaces  used  for  communication,  and  uses  many 
commercial  off  the  shelf  (COTS)  components,  which  facilitated  infonnation  gathering.  As  this 
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system  is  fictional,  data  for  the  ACIH  could  be  quickly  generated.  An  executive  summary  of  the 
ACIH  is  provided  in  Appendix  D. 


c.  Creating  A  rtifacts 

Artifacts  are  required  to  assess  I/ORLs  and  to  perform  a  SETR  review.  For 
implementing  the  TTX  at  the  reviews,  the  team  noted  entrance  criteria  outlined  in  the  SETR 
[SETR,  2009]  and  other  additional  documents/graphics  to  determine  the  necessary  artifacts.  The 
artifacts  created  for  the  ASR  includes:  an  AoA,  High  Level  Operational  Concept  Graphic  (OV- 
1),  Risk  Assessment/Risk  Management  Plan,  requirements,  a  notional  schedule,  and  a 
Technology  Development  Plan.  The  artifacts  created  for  the  OTRR  includes:  Operational 
Requirements  Document  (ORD),  Operational  Test  Plan,  DT&E  Results  and  test  report,  and 
Training  Plan.  These  artifacts  are  provided  in  Appendix  D. 

d.  Planning  for  the  I/ORL  Process  TTX 

All  participants  of  the  TTX  were  of  members  from  the  overall  project  team.  The  TTX 
team  was  separated  into  the  following  three  categories  with  their  respective  responsibilities: 

•  Scripter/Facilitator 

Design  the  table  top  exercise  scenarios  and  script 

-  Lead  discussions 

•  Role  Players 

-  Perfonn  the  process  using  procedures  and  protocols 
React  to  situations 

•  Observers 

Take  notes  on  the  events 

Provide  overall  suggestions  for  improvements 

These  key  participants  were  required  to  review  the  FORL  Process  and  perform  the 
necessary  steps  as  outlined  in  Chapter  IV  of  this  report.  Participants  were  reminded  that  the  goal 
was  to  assess  the  I/ORL  for  each  interface  in  the  context  of  executing  the  mission  thread.  The 
role  players  required  for  the  ACIH  TTX  were: 

•  SETR  Lead/Board 

Responsible  for  conducting  the  review  and  is  the  decision  authority  for  the  pass/fail 
judgment  of  the  interfaces. 

SETR  Handbook  documents  exactly  how  the  SETR  lead  is  identified  for  each  review. 

•  System  Integrator 

The  System  Integrator  is  responsible  for  integrating  the  system  to  ensure  readiness  for 
OT.  They  will  ensure  the  system  is  ready  for  integration  testing  by  providing 
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feedback  to  the  SMEs  at  each  iterative  review.  They  can  also  provide  testing  results 
to  the  SMEs  to  help  assess  EORLs  at  certain  reviews. 

Within  the  Army,  the  System  Integrator  would  be  identified  from  Program  Executive 
Office  Soldier  (PEO  Soldier).  The  System  Integrator  role  can  also  be  contracted  out 
by  the  PEOs. 

•  System/Element  SMEs 

SMEs  are  identified  from  the  owners  of  the  system.  SMEs  can  either  be  Government 
or  contractor  personnel.  Example  organizations  include:  In-Service  Engineering 
Agent,  Developer,  Contracted  support,  etc. 

-  ACIH  Mission  Thread  Specific:  Display,  Core  HW/SW,  and  Audio/Video 

•  Operational  Test  Agency  Representative 

The  representative  is  responsible  preparing  the  Operational  tests.  The  representative 
will  also  ensuring  the  system  is  testable  during  the  OT  process  by  providing  feedback 
to  the  SMEs  at  each  iterative  review. 

The  representative  would  come  from  COMOPTEVFOR 

•  3rd  Party  Representative 

The  representative  would  be  responsible  for  fostering  SME  discussion  and  to  support 
the  assigning  I/ORL  values  as  an  unbiased  3  rd  party. 

The  representative  is  assigned  by  the  program  office  and  can  be  either  a  contractor  or 
government  personnel. 


Due  to  time  constraints,  the  TTX  was  designed  to  explore  only  one  mission  thread.  The 
team  selected  the  ACIH  System  Mission  Thread  pertaining  to  audio  and  visual  communication. 
This  thread  integrates  various  audio,  visual,  and  system  components  within  the  helmet  to  a  core 
computing  system.  The  helmet  will  enable  the  user  to  communicate  with  other  ground  troops 
and  headquarters  both  visually  and  verbally  through  direct  air  transmission  and/or  via  satellite 
communication,  increasing  the  user’s  situational  awareness.  This  thread  must  include  the 
organizations  responsible  for  fielding  the  equipment,  and  the  organizations  responsible  for 
ensuring  the  logistical  infrastructure  to  support  the  use  of  the  helmet  throughout  the  lifecycle. 
This  thread  was  chosen  because  many  of  the  components  used  within  the  thread  are  COTS, 
allowing  the  team  to  easily  find  data. 

e.  Goals  of  the  TTX 

The  TTX  addressed  the  following  questions  about  the  I/ORL  Process: 

•  How  effective  is  the  new  process  at  ensuring  programs  pass  Operational  Testing? 

•  What  areas  in  the  I/ORL  Process  are  missing  or  require  more  definition  and  clarification? 
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2.  Performing  the  Table  Top  Exercise  -  ASR 

The  team  performed  the  I/ORL  Process  using  the  ASR  as  the  first  review;  however,  some 
procedures  were  not  executed  as  expected.  This  resulted  in  the  team  discussing  issues  and 
generating  solutions  following  the  exercise. 

a.  Performing  the  I/ORL  Process 

The  team  began  the  I/ORL  Process  by  defining  the  given  Mission  Critical  Thread.  The 
SMEs  attempted  to  discuss  what  interfaces  were  relevant.  The  following  is  a  list  of  interfaces 
selected: 

•  Audio  and  video  needs  to  be  encrypted 

•  Communications  prevent  j  amming 

•  Coverage  to  satellite/SATCOM 

•  Encryption  key  needs  to  change 

•  The  software  should  interface  all  individual  components  together  and  provide 

communication 

•  Requirement  for  3  dimensional  display 

As  the  Mission  Critical  Thread  was  not  clearly  defined  by  the  team,  the  team  improperly 
identified  interfaces  for  the  system.  The  team  complicated  matters  further  by  discussing  where 
the  program  should  be  in  respect  with  the  ASR,  versus  using  the  given  artifacts  to  help  identify 
interfaces.  While  the  items  in  the  list  above  are  not  actually  interfaces,  the  discussions  that 
ensued  occasionally  highlighted  actual  interfaces,  such  as  the  hardware-to-hardware  interface 
between  the  helmet  and  the  satellite,  and  the  software-to-hardware  interface  between  the  core 
software  and  other  various  components. 

With  the  team  in  technical  discussions  on  these  identified  “interfaces”,  the  team  began 
assigning  I/ORL  values  to  each  interface  based  on  the  LORL  Values  as  defined  in  Chapter  IV  of 
this  paper.  These  values  were  then  compared  to  the  I/ORL  threshold  and  objective  value 
requirement  for  the  review.  If  an  LORL  did  not  meet  its  objective  as  outlined  in  Table  10,  a  risk 
mitigation  plan  was  generated.  The  assessed  values  and  required  mitigation  steps  were  later 
presented  to  the  SETR  Lead  and  OTA  Representative  for  approval.  The  results  are  presented  in 
Table  14. 
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Table  14. 


ASR  TTX  Results 


This  table  shows  the  assessed  I/ORL  values  for  the  ACIH  System  during  the  ASR  TTX. 


Interface 

I/ORL 

Objective 

I/ORL 

Threshold 

I/ORL 

Assessed 

Value 

Mitigation 

Step 

Pass/Fail 

Encrypt 

audio/video 

data 

2 

1 

2 

N/A 

Pass 

Anti-jamming 

2 

1 

1 

Do  more 
research 

Pass  with 
mitigation 

Satellite 

communications 

2 

1 

2 

N/A 

Pass 

Power 

2 

1 

2 

N/A 

Pass 

Software 

2 

1 

2 

N/A 

Pass 

3D  Display 

2 

1 

2 

N/A 

Pass 

b.  Assessing  the  ASR  TTX 

Once  the  TTX  was  completed,  the  assigned  facilitators  and  observers  discussed  their 
comments  on  the  exercise.  Issues  noted  upon  completion  of  the  TTX  for  the  ASR  included  an 
unclear  definition  of  Mission  Critical  Threads,  unclear  definition  of  an  interface,  and  a  failure  to 
identify  critical  interfaces.  Additionally,  I/ORL  ranking  levels  were  vague  and  no  procedure 
existed  for  ranking  I/ORLs.  Finally,  cost  applicability  and  goals  of  reviews  needed  to  be 
outlined.  The  following  solutions  were  generated  to  clarify  the  process  based  on  lessons  learned 
from  the  ASR: 

Mission  Criticality  Thread  Representation 

Issue :  A  major  issue  from  the  table  top  exercise  was  detailing  how  to  represent  the 
mission  thread  topic.  The  exact  definition  of  mission  criticality  was  not  understood  and  the 
method  in  representing  the  mission  thread  in  a  manner  to  help  the  LORL  Process  caused 
confusion  among  the  team. 

Solution-.  In  order  to  remedy  this  issue,  appropriate  diagrams  are  required  to  define  and 
accurately  represent  the  mission  critical  thread.  A  flow  chart  can  be  used  as  an  important  tool  to 
describe  the  mission  thread.  It  can  visually  show  how  mission  threads  flow  from  start  to  finish. 
This  allows  the  team  to  identify  critical  system  interfaces  to  be  used  in  the  I/ORL  Process.  These 
interfaces  can  then  be  assessed  during  criticality  assessment.  The  flow  chart  becomes  a  valuable 
tool  in  representing  the  mission  thread  to  help  identify  the  interfaces.  A  top  level  flow  chart  can 


96 


become  the  main  diagram,  which  can  further  refined  as  other  diagrams  such  FFBDs  and  IDEFOs 
emerge. 

Interface  Definition 

Issue:  During  the  table  top  exercise,  it  was  realized  that  interface  definition  was  not 
clearly  characterized. 

Solution:  To  address  the  lack  of  an  interface  definition  that  was  identified  during  the 
table  top  exercise,  the  team  must  define  interfaces  associated  with  the  mission  thread  and  assess 
the  FORL.  Defining  each  interface  of  a  system  prior  to  and  during  the  interface  design  process 
is  essential  to  the  success  of  system  interoperability.  Interface  definition  includes  defining 
external  interfaces  with  other  systems,  defining  interfaces  of  system  end  items  with  each  other, 
and  defining  interface  objectives. 

The  mission  thread  flow  chart  and  other  available  diagrams  can  be  used  to  help  identify 
interfaces.  These  interfaces  can  be  human,  hardware,  and  software  aspects  of  a  system.  These 
interfaces  can  be  categorized  as  software-to-software,  software -to-hardware,  and  hardware-to- 
hardware. 

Critical  Interface  Identification 

Issue:  Key  players  failed  to  identify  critical  interfaces  as  part  of  the  FORL  Process. 

Solution:  While  all  interfaces  are  important  in  the  SE  process,  certain  interfaces  must  be 
distinctly  considered  to  carry  out  the  FORL  Process.  The  critical  interfaces  should  be 
highlighted  and  brought  to  attention  in  the  program  review.  The  focus  on  critical  interfaces  prior 
to  the  review  will  allow  the  system  designer  and  developer  to  closely  plan  and  lay  out  the  system 
interfaces  to  satisfy  the  mission  requirements.  Early  consideration  and  planning  of  critical 
interfaces  will  make  certain  that  system  interoperability  is  integrated  into  the  design  and 
development  of  the  system. 

Prior  to  the  review,  a  team  of  engineers  are  required  to  identify  critical  interfaces  for  a 
critical  mission  thread  in  order  to  test  the  FORL  Process.  The  engineers  identified  the  applicable 
interfaces  for  the  critical  mission  thread  but  failed  to  identify  the  interfaces  that  have  the  most 
impact  in  the  thread,  the  critical  interfaces.  They  did  not  assess  the  interface  criticality  prior  to 
assigning  the  FORL  objective  and  threshold.  The  mistake  was  discovered  when  the  engineers 
had  difficulty  assigning  FORL  values  to  an  interface  that  was  not  critical  in  the  FORL  Process. 
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If  the  non-critical  interfaces  had  been  identified  earlier,  the  team  would  have  recognized  that  the 
interfaces  did  not  have  significant  roles  in  the  process. 

I/ORL  Rankins 

Issue:  The  definitions  for  each  I/ORL  as  described  in  Chapter  IV  of  this  report  are  not 
universal  and  it  is  difficult  to  apply  to  various  interfaces. 

Solution:  One  way  to  clarify  the  I/ORL  descriptions  and  make  them  more  universal  is  to 
provide  a  brief,  high-level  summary  of  the  I/ORL  ranks  to  be  used  in  addition  to  the  detailed 
explanations.  It  is  important  to  note  that  the  brief  descriptions  are  useful  for  clarification  of 
I/ORL  values,  however,  the  complete  definition  of  an  I/ORL  must  be  used  when  making  as 
assessment.  This  summary  of  I/ORL  values  is  outlined  in  Table  15. 


Table  15.  High-Level  Description  of  I/ORL  Values 

This  table  describes  the  brief,  high-level  summaries  developed  for  each  of  the  I/ORL  values. 


I/ORL 

High-Level  Description 

9 

In  Operational  Use 

8 

Passed  OT&E 

7 

Component  Interoperability  Successful  &  Operational  Scenarios  Simulated 

6 

Passed  DT&E 

5 

Component  Design  Complete  &  Reviewed  for  Interoperability  Deficiencies 

4 

Technology  Demonstration  of  Interfaces 

3 

Interfaces  Requirements  Documented 

2 

AoA  Addressing  Interoperability  Complete 

1 

Identification  and  Exploration  of  Interfaces 

Another  solution  to  help  clarify  I/ORL  levels  is  to  develop  a  training  program.  Although 
the  LORL  Process  is  fairly  simple,  understanding  I/ORL  definitions  is  not  trivial.  A  curriculum 
could  be  established  that  expands  on  these  definitions  and  provides  examples  on  how  to  use 
them.  This  could  be  offered  via  the  Defense  Acquisition  University  as  an  online  class. 

Having  more  tools  available  to  aid  in  understanding  will  allow  those  involved  in  the 
LORL  Process  to  more  accurately  assess  LORL  values  for  interfaces.  This  in  turn  will  help  the 
interfaces  and  consequently  the  system  to  perform  better  during  operational  testing. 

I/ORL  Rankins  Procedure 

Issue:  No  procedure  was  provided  in  the  LORL  Process  for  ranking  I/ORLs. 
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Solution:  A  clearly  defined  process  for  ranking  or  assigning  values  to  interfaces  would 
assist  the  team  in  the  I/ORL  Process.  By  outlining  the  ranking  procedure,  evaluators  have  a  clear 
"means  of  attack"  and  can  more  accurately  assess  I/ORL  values  for  interfaces.  Ultimately  this 
will  help  the  system  interfaces  to  be  more  robust  during  operational  test  and  evaluation. 

Once  data  is  available  to  detennine  the  maturity  of  critical  interfaces,  I/ORL  values  are 
assigned  to  each  interface  as  part  of  the  I/ORL  Process.  In  order  to  assign  I/ORL  values  in  a 
consistent  way,  a  ranking  procedure  must  be  developed.  The  process  for  ranking  I/ORLs  is 
outlined  briefly  below: 

•  Review  Interoperability  Assessment  Data:  Prior  to  a  review,  subject  matter  experts 
have  detennined  Critical  System  Interfaces  as  outlined  in  Chapter  IV  of  this  report. 

These  are  the  interfaces  on  which  an  I/ORL  assessment  is  to  be  made.  SMEs  will 
provide  technical  specifications  required  to  assess  the  CSIs.  Personnel  responsible  for 
assigning  I/ORL  values  must  review  this  data. 

•  Understand  I/ORL  Values:  Evaluators  are  responsible  to  ensure  they  understand  the 
meaning  of  I/ORL  values.  These  values  are  described  in  Chapter  IV  in  the  section  titled 
"I/ORL  Values".  Additionally,  Table  15  is  provided  with  increasing  I/ORL  values  and 
hence  maturity  is  presented.  Both  sets  of  data  can  be  used  to  understand  the  meaning  of 
each  I/ORL  value.  Evaluators  should  attend  training  if  it  is  available. 

•  Discuss  Maturity  of  Interface:  All  parties  involved  in  assigning  interfaces  must 
thoroughly  discuss  critical  interfaces.  The  team  must  fully  confer  about  the  LORL  data 
in  order  to  fully  comprehend  the  level  of  maturity  of  each  interface. 

•  Assess  Maturity  of  Interface:  Once  discussion  has  occurred  evaluators  must  assess  the 
interoperability  data  presented  by  SMEs.  Evaluator  should  assess  the  maturity  or 
interoperability  of  each  interface  bearing  in  mind  the  LORL  values  and  their  meaning. 

•  Assign  I/ORL  Values:  Upon  assessing  the  interoperability  maturity  of  an  interface, 
evaluators  assign  an  LORL  based  on  the  designated  values  in  Chapter  IV. 

•  Repeat  For  Each  Interface:  Repeat  this  process  for  each  interface  until  I/ORLs  have 
been  assigned  to  each  CSI. 


Cost  Application 

Issue:  The  process  of  allocation  of  cost  with  respect  to  system  interfaces  and  their 
interoperability  within  the  overall  system  design  from  programmatic,  life  cycle  cost  (LCC),  and 
implementation  viewpoints  was  unclear. 

Solution:  There  are  many  factors  in  the  SE  process  and  cost  should  always  be  a 
consideration  during  this  process.  The  LORL  Process  can  be  used  to  help  focus  on 
interoperability  areas  that  are  known  to,  or  could  present  possible  cost  application  risks  to  a 
system’s  design  and  development.  Program  engineers  and  budget  analysts  need  to  consider  the 
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various  stages  of  the  I/ORL  Process  and  evaluate  the  cost  associated  with  ensuring  that  the 
interoperability  of  the  system  is  in  line  with  the  system’s  respective  stage  of  development. 

Implementation  of  bringing  an  interface  to  the  appropriate  I/ORL  level  can  present  a 
severe  strain  on  the  overall  budget  of  a  program  if  programmatic  cost  and  LCC  are  not  kept  in 
check.  Programmatic  costs  can  seem  small  at  first,  but  these  costs  must  be  reviewed  and  vetted 
through  the  whole  program  to  capture  the  true  nature  of  how  much  it  might  cost  to  mitigate  a 
potentially  low  I/ORL  during  a  review  cycle.  This  cost  does  not  just  stop  with  trying  to  mitigate 
an  interface  to  the  next  appropriate  interoperability  readiness  level,  but  can  extend  well  beyond 
the  system  engineering  and  design  phase.  The  cost  application  should  be  considered  even  into 
the  out  years  with  the  entire  life  cycle  cost  for  the  program  due  to  modifications  made  during  the 
early  development  stages. 

The  table  top  exercise  demonstrated  with  the  ranking  of  interfaces  with  I/ORLs  where  the 
interfaces  were  not  at  the  threshold  LORL  value  for  the  associated  review;  cost  considerations  to 
get  the  interface  to  the  threshold  level  should  be  thoroughly  investigated  by  the  PM.  The  PM 
must  then  weigh  all  the  factors  involved  to  determine  the  way  forward.  This  type  of  exercise  can 
give  great  insight  into  how  the  design  team  and  program  managers  should  work  hand  in  hand 
during  early  development  to  ensure  the  right  systems  and  interfaces  are  getting  the  correct 
attention,  and  the  system  is  progressing  at  a  suitable  pace  so  as  to  avoid  interoperability  failures 
later  in  the  operational  testing  phases. 

Goals  of  Review 

Issue:  The  Alternative  System  Review  (ASR)  is  a  technical  review  that  demonstrates  the 
preferred  concept  is  cost  effective,  affordable,  operationally  effective  and  suitable;  and  can  be 
developed  to  provide  a  timely  solution  at  an  acceptable  level  of  risk.  The  ASR  ensures  that  the 
resulting  set  of  requirements  satisfies  the  customers'  needs  and  expectations,  the  system’s 
concepts  align  with  the  external  environment,  and  the  system  under  review  is  mature  enough  to 
proceed  into  the  Technology  Development  phase.  These  are  the  textbook  definitions  as  defined 
in  the  Defense  Acquisition  Guidebook  (DAG)  but  the  goals  are  not  clear  in  terms  of  interfaces 
(hardware  and  software)  or  Interoperability  Readiness  Levels  (I/ORLs). 

Solution:  The  ASR  provides  an  agreement  on  the  proposed  material  solutions,  hardware 
and  software  architectural  constraints  or  drivers  to  address  all  key  perfonnance  parameters 
(KPPs).  As  assessment  of  the  full  system  software  concept,  a  comprehensive  rationale  for  the 
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proposed  material  solutions,  a  comprehensive  assessment  of  the  relative  risks,  and  a 
comprehensive  risk  assessment  for  the  Technology  Development  phase  are  performed.  The  ASR 
also  provides  results  of  trade  studies,  technical  demonstrations,  joint  requirements,  refined 
thresholds  and  objectives,  and  a  draft  system  requirements  document  all  in  terms  of  cost 
effectiveness,  affordability,  and  providing  an  acceptable  level  of  risk.  During  the  ASR,  key 
system  components  and  interfaces  (both  physical  and  functional)  need  to  be  identified  across  the 
entire  system  and  interface  design.  As  seen  during  the  TTX,  I/ORLs  need  to  be  assigned  to 
system  interfaces  where  I/ORL  objectives  and  thresholds  are  identified.  During  the  ASR,  I/ORL 
values  are  assessed  to  the  interfaces  and  if  the  I/ORL  threshold  is  not  met  then  mitigation  steps 
are  identified.  This  will  help  identify  any  issues  in  regard  to  interfaces  and  system  requirements 
prior  to  going  to  the  Technology  Development  phase.  Changing  requirements  or  interfaces  later 
in  the  program  development  will  usually  produce  cost  increases  and  schedule  slips.  The  overall 
goals  of  the  ASR  should  be  entrance  criteria  for  a  system  going  into  the  Technology 
Development  phase  and  System  Requirements  Review  (SRR). 

3.  Performing  the  Table  Top  Exercise  -  OTRR 

This  review  ensures  that  the  production  configuration  system  can  proceed  into  Initial 
Operational  Test  and  Evaluation  (IOT&E)  with  a  high  probability  of  success.  The 
interoperability  goals  of  the  OTRR  are  to  verify  interoperability  of  the  system  with  all  other 
systems  has  been  verified  through  laboratory  testing,  simulated  operational  scenarios,  ground 
based  testing,  and  under  operational  environmental  testing  (desert  urban  environment  in  the  day 
or  night  using  military  special  forces  executing  a  planned  representative  task  or  rescue  mission) 
(LORL  7). 

a.  Performing  the  I/ORL  Process 

The  I/ORL  Process  was  performed  by  the  team  while  leveraging  off  the  lessons  learned 
from  the  ASR  TTX.  The  OTRR  focused  on  Developmental  Test  and  Evaluation  (DT&E)  results 
of  the  ACIH  program,  a  provided  artifact.  The  team  reviewed  the  goals  of  the  OTRR  to  ensure 
the  team  shared  a  common  general  understanding  of  the  objectives  and  to  ensure  a  successful 
outcome  of  the  review. 
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Before  starting  the  TTX,  the  team  regenerated  the  Mission  Critical  Thread  definition  and 
the  respective  Critical  System  Interfaces.  The  Mission  Critical  Thread  definition  was  supported 
by  a  diagram,  presented  in  Figure  28. 
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Figure  28.  Communications  Mission  Thread  for  the  ACIH  System 

This  figure  depicts  the  communications  mission  thread  for  the  fictional  ACIH  System.  The  left  side  of  the  diagram 
represents  the  main  internal  components  of  the  ACIH  required  for  the  mission  thread  and  organizations  that  support 
the  ACIH.  The  right  side  of  the  diagram  represents  the  external  components  that  the  ACIH  must  be  interoperable 

with:  satellite,  T/R  Headquarters,  and  other  Helmet  T/Rs. 


Using  this  diagram,  the  team  was  able  to  identify  interfaces  within  the  mission  thread  and 
discuss  which  interfaces  were  critical.  The  following  is  a  list  of  the  CSIs: 

•  Microphone-to-Core  SW/HW 

•  Speakers-to-Core  SW/FIW 

•  Core-SW/FIW-to  Display 

•  Core  SW/HW-to-Transmit/Receive  (T/R) 

•  T/R-to-Satellite 

•  Satellite-to-T/R  Headquarters 

•  User  Interface-to-Core  SW/HW 

•  User-to-User  Interface 

•  Support  Activity-to-HW  Equipment 

•  Support  Activity-to-Software 

•  Support  Activity-to-Developer 
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•  OS-to-Encryption  SW 

•  User-to-Headquarters 


The  interfaces  above  identify  the  various  hardware  and  software  components  within  the 
helmet  that  must  be  interoperable  to  ensure  the  fulfillment  of  the  mission  thread.  Also  identified 
are  the  various  organizations/people  vital  to  using,  developing,  and  maintaining  the  hardware  and 
software  throughout  the  helmet’s  life  cycle. 

Experience  from  previously  completing  the  ASR  TTX  essentially  provided  training  the 
team  required.  This  resulted  in  a  smoother  execution  of  the  process.  The  team  referred  to  the 
process  and  artifacts  more,  and  utilized  the  revised  I/ORL  procedure  to  help  assign  I/ORL 
values.  This  resulted  in  less  confusion  in  assigning  I/ORL  values,  and  allowed  the  team  to 
effectively  find  issues  with  the  system. 

Upon  completing  the  improved  I/ORL  Process,  the  team  generated  interface  assessments, 
evaluated  these  assessments,  and  issued  a  Pass  or  Fail  designation.  Results  are  presented  in 
Table  16. 


Table  16.  OTRR  TTX  Results 


This  table  shows  the  assessed  I/ORL  values  for  the  ACIH  System  during  the  OTRR  TTX. 


Interface 

I/ORI. 

Objective 

I/ORL 

Threshold 

I/ORL 

Assessed 

Value 

Mitigation  Step 

Pass/Fail 

Microphone-to-Core 

SW/HW 

7 

7 

7 

N/A 

Pass 

Speakers-to-Core 

SW/HW 

7 

7 

6 

Re-evaluate  requirement  for 
audio  frequency/Find  cause 
of  low  frequency  and  conduct 
additional  testing 

Fail 

Core-SW/HW-to 

Display 

7 

7 

6 

Re-evaluate 
requirement/investigate 
display  component 

Fail 

Core  SW/HW-to- 
Transmit/Receive 
(T/R) 

7 

7 

7 

N/A 

Pass 

T/R-to-Satellite 

7 

7 

6 

Investigate  45  second 
delay  issue. 

Fail 

Satellite-to-T/R 

Headquarters 

7 

7 

6 

Investigate  45  second 
delay  issue. 

Fail 

User  Interface-to- 
Core  SW/HW 

7 

7 

7 

N/A 

Pass 
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Interface 

I/ORL 

Objective 

I/ORL 

Threshold 

I/ORI. 

Assessed 

Value 

Mitigation  Step 

Pass/Fail 

User-to-User  Interface 

7 

7 

6 

Perfonn  DT  user-to-user 
interface  before  OTRR 

Fail 

ISEA-to-HW 

Equipment 

7 

7 

7 

N/A 

Pass 

ISEA-to-Software 

7 

7 

7 

N/A 

Pass 

ISEA-to-Developer 

7 

7 

7 

N/A 

Pass 

OS-to-Encryption  SW 

7 

7 

7 

N/A 

Pass 

User-to-Headquarters 

7 

7 

6 

This  issue  is  dependent  on 
other  issues  to  be  resolved. 

Fail 

As  shown  within  the  results,  the  TTX  was  successful  in  identifying  critical  system 
interfaces  not  ready  for  Operational  Testing.  By  identifying  these  failures  ahead  of  time,  the 
I/ORL  Process  will  have  prevented  a  failed  Operational  Test. 

b.  Assessing  the  OTRR  TTX 

The  TTX  for  OTRR  benefitted  greatly  from  the  initial  ASR  TTX.  However,  while 
performing  the  TTX  the  team  discovered  a  few  minor  issues;  one  dealing  with  the  tracking  table 
used  during  the  review,  and  the  other  concerning  timing  of  events. 

Additional  Column:  Mitigation  Step  Accept/Disagree 

Issue :  While  the  team  performed  the  second  phase  of  the  TTX,  the  team  discovered  one 
minor  issue  in  the  table  used  to  keep  track  of  the  interfaces.  When  the  SETR  Board  was 
deciding  whether  or  not  the  mitigation  step  was  acceptable  or  not,  there  was  no  column  to  input 
their  decision. 

Solution :  An  additional  column  is  required  to  keep  track  of  whether  or  not  the  mitigation 
step  is  acceptable  to  the  SETR  Board.  This  is  an  important  option  for  the  SETR  Board  in  case 
the  suggested  mitigation  step  is  unacceptable  due  to  cost,  schedule,  or  relevance.  The  mitigation 
step  can  be  updated  and  accepted  at  the  review  or  at  a  later  time. 

Time  Between  I/ORL  Assessment  and  the  SETR 

Issue :  During  the  discussions  of  mitigation  steps  with  the  SETR  Lead  and  the 
OPTEVFOR  Representative,  one  concern  that  was  brought  up  was  that  the  time  between  the 
I/ORL  assignment  meeting  and  the  actual  technical  review  was  not  defined  in  the  process.  The 
concern  was  brought  up  when  minor  ACIH  issues  from  the  first  phase  (I/ORL  Meeting)  were 
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discussed  in  the  second  phase,  questioning  whether  issues  found  would  be  fixed  by  the  time  the 
review  occurs. 

Solution :  While  the  amount  of  time  between  the  assessment  meeting  and  the  review  is 
ultimately  detennined  by  the  program  manager  based  on  the  status  of  the  program,  the  team 
suggests  the  meeting  be  held  anywhere  from  a  month  to  two  weeks  before  the  actual  review. 

This  will  allow  enough  time  to  mitigate  any  small  issues  found  during  the  assessment  meeting. 
However,  any  major  issues  costing  the  program  additional  funds  and/or  affecting  the  schedule  of 
the  system  must  be  approved  by  the  SETR  Board. 

4.  Summary  of  Table  Top  Exercise  Results 

The  goal  was  to  perform  the  TTX  at  different  sections  of  the  System  Engineering  Process 
and  gradually  improve  the  process  as  the  authors  progressed  through  the  reviews.  The  I/ORL 
Process  was  assessed  by  going  through  an  ASR  and  OTRR  for  a  fictitious  program.  Upon 
completion  of  validating  the  EORL  Process  via  a  TTX,  several  valuable  lessons  emerged  from 
this  exercise. 

a.  The  Need  for  Training 

The  I/ORL  Process  is  most  effective  when  participants  fully  understand  the  process  and 
are  able  to  use  their  knowledge  and  skills  to  apply  it  to  a  program.  All  involved  personnel 
should  have  a  common  understanding  of  the  I/ORL  Process  to  collectively  apply  it  to  a  system. 
It  is  important  that  the  participants  are  provided  with  training  and  guidelines  to  increase  their 
understanding  and  improve  the  application  of  the  process. 

The  need  for  training  was  highlighted  by  the  difference  in  performing  the  I/ORL  Process 
between  the  ASR  and  the  OTRR.  In  the  ASR,  the  team  neglected  to  identify  critical  interfaces. 
The  lack  of  understanding  the  I/ORL  Process  among  key  participants  led  to  confusion  and 
lengthy  program  reviews.  Performing  the  ASR  effectively  provided  the  team  training.  With  a 
greater  understanding  of  the  process  and  I/ORL  values,  the  team  executed  more  efficiently.  The 
OTRR  TTX  demonstrated  that  improvement  in  understanding  the  I/ORL  Process  enhanced  and 
expedited  execution. 
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b.  Skipping  a  Review  Can  Lead  to  Problems 

Performing  the  TTX  proved  challenging  due  to  the  fact  that  the  team  only  performed  two 
reviews  to  validate  the  I/ORL  Process.  The  team  noticed  that  issues  found  within  the  OTRR 
TTX  would  have  been  caught  in  earlier  reviews.  If  the  TTX  consisted  of  more  SETR  reviews 
earlier  in  the  system  lifecycle,  these  issues  would  likely  have  been  caught  earlier,  resulting  in 
fewer  issues  at  OTRR. 

This  TTX  lessons  learned  can  be  applied  to  the  I/ORL  Process.  It  demonstrates  the  need 
to  apply  the  I/ORL  Process  to  all  the  relevant  reviews  throughout  the  lifecycle  of  a  system.  If  the 
I/ORL  Process  is  skipped,  it  is  possible  that  the  system  will  progress  through  reviews  with  some 
immature  interfaces.  This  could  in  turn  lead  to  a  system  failing  operational  testing. 

c.  The  I/ORL  Process  Worked 

After  conducting  the  OTRR,  the  team  found  critical  interfaces  that  were  not  at  the  proper 
I/ORL  in  order  to  proceed.  The  team  found  that  the  I/ORL  Process  worked  well  with  regard  to 
identifying  interoperability  issues  before  the  system  proceeds  to  formal  Operational  Test.  If 
these  issues  had  not  been  caught  by  applying  the  I/ORL  Process  to  the  system,  the  system  would 
have  proceeded  into  OT&E  and  would  likely  have  failed  at  some  point.  This  would  have  had  a 
major  impact  on  cost  and  schedule  for  the  program  and  would  have  ended  in  considerable 
amounts  of  follow-on  testing  in  order  to  verify  and  secure  the  proper  corrections 

By  perfonning  the  TTX  at  two  reviews,  the  team  validated  that  the  I/ORL  Process  is 
effective  at  ensuring  that  programs  will  likely  pass  operational  testing.  Further,  the  team 
identified  areas  in  the  I/ORL  Process  that  require  more  definition  and  clarification,  and  areas  that 
were  missing. 

B.  MODELING  AND  SIMULATION 

The  second  part  of  the  validation  is  a  mathematical  and  computer  model  designed  to 
demonstrate  the  high-level  effectiveness  of  an  I/ORL  implementation.  The  model  stochastically 
simulates  the  interoperability  work  done  throughout  the  SE  process  by  assigning  I/ORL  values  to 
system  components  and  compares  the  assigned  value  to  the  required  values  at  each  of  the 
decision  gates.  If  the  system  does  not  meet  the  prescribed  I/ORL  threshold  for  the  technical 
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review,  the  system  is  reworked  to  meet  the  threshold  at  an  additional  cost  that  is  assessed 
according  to  the  development  phase. 

If  the  system  reaches  operational  test  and  does  not  meet  the  prescribed  I/ORL  threshold, 
the  system  is  deemed  to  have  failed.  This  process  is  automated  and  repeated  for  a  large  number 
of  systems  and  the  number  of  failures  is  counted  as  well  as  total  cost  of  failures  incurred  to  the 
program. 

To  generate  a  meaningful  comparison,  the  simulation  performs  two  parallel  models:  one 
of  the  current  SE  process  and  one  of  the  modified  I/ORL  Process.  Prior  to  each  review,  the 
interoperability  is  increased  for  each  system  in  both  models.  For  the  new  process,  if  the  system 
does  not  meet  the  I/ORL  threshold  values,  it  is  reworked  until  it  matches  the  threshold. 

Ultimately,  the  model  takes  random  inputs  simulating  the  size  of  the  program  and  data 
outlining  the  effectiveness  of  the  current  SE  process,  and  outputs  relative  OT  pass  rates  and  cost 
impacts.  The  simulation  compares  the  results  of  the  two  processes  to  draw  conclusions  on  the 
effectiveness  of  adding  the  I/ORL  measurement  process.  See  Figure  29  for  a  generic  flow 
diagram  for  the  simulation: 


I/ORL  Process 


Current  SE  Process 


$ 


Figure  29.  Flow  Diagram  for  Modeling  and  Simulation 

This  diagram  shows  the  basic  model  structure.  Each  system  (represented  by  the  red  dot),  moves  through  a  series  of 
decision  gates.  As  the  system  moves  across  each  arrow  in  the  diagram,  the  system  is  being  worked  on  and  the 
interoperability  improves.  In  the  I/ORL  Process,  the  system  is  evaluated  at  each  of  the  technical  reviews  (A,  B,  and 
C  in  the  above  diagram),  and,  if  the  system  does  not  meet  the  review  requirements,  it  is  reworked  until  it  does. 
During  the  current  SE  process,  the  interoperability  is  not  rigorously  and  consistently  evaluated  against  a  defined 
criteria  and  the  system  never  incurs  any  rework.  At  OT,  the  system  has  its  interoperability  field  tested  and  it  is 

determined  to  either  pass  or  fail. 
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1.  Model  Assumptions 

Given  both  the  time  constraints  and  available  data,  the  authors  made  the  following 
assumptions  to  facilitate  the  model  creation  process.  The  authors  made  the  detennination  based 
on  engineering  judgment  that  none  of  the  assumptions  made  would  significantly  alter  the 
conclusions  drawn  by  the  model  in  a  negative  manner. 

•  The  maturity  of  an  interface’s  design  can  accurately  be  represented  by  a  single  number 
reflective  of  the  interface’s  interoperability 

a.  Interoperability  can  be  modeled  as  a  real  number  despite  I/ORLs  only  being 
integers. 

b.  For  the  purposes  of  this  model,  the  maturity  of  an  interface  design  is  represented 
by  a  single  number  on  an  interval  scale,  which  is  a  constraint  on  those  values  not 
included  in  the  I/ORL  definitions  provided  earlier. 

•  Asa  system  design  matures,  the  work  done  between  technical  reviews  is  represented  by 
an  increase  in  the  I/ORL  level.  That  increase  will  not  always  be  the  same  for  each 
interface  and  nor  from  system  to  system.  Therefore,  a  random  number  generator  was 
included  in  the  LORL  model  between  simulated  reviews  to  represent  design  maturation 
in  a  realistic  way.  Furthermore  this  assumes  that 

o  There  is  an  equal  probability  to  be  above  or  below  the  mean 
o  Interoperability  work  is  well  behaved,  i.e.  the  standard  deviation  is  significantly 
smaller  than  the  mean 

•  The  total  amount  of  interoperability  work  is  the  same  between  each  review 

•  The  cost  incurred  to  correct  a  design  flaw  is  a  fixed  cost  based  on  the  milestone  at  which 
the  problem  was  discovered 

•  The  amount  of  interoperability  tracking  in  the  current  SE  process  is  minimal 

2.  Model  Structure 

The  model  is  a  procedural  program  written  in  the  MATLAB  language.  MATLAB  was 
chosen  for  its  familiarity  to  the  authors  and  due  to  its  ease  of  processing  array  operations.  The 
full  code  is  included  in  Appendix  E. 

The  model  is  designed  to  run  a  series  of  evaluations  on  a  single  system  and  record  the 
results.  That  process  is  repeated  a  number  of  times  (-10,000)  and  statistics  are  gathered  from  the 
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results.  As  mentioned  above,  the  model  runs  two  concurrent  models,  one  for  the  current  SE 
process  and  one  for  the  modified  SE  process,  on  an  identical  system. 

The  model  sequence  is  broken  down  into  the  following  phases: 

•  Data  Initialization 

•  Interoperability  Work 

•  I/ORL  Measurement 

•  Review  Decision 

•  Interoperability  Rework 

•  Operational  Test  Evaluation 

•  Data  Storage 

The  description  of  each  phase  follows. 

a.  Data  Initialization 

During  the  data  initialization  phase,  the  simulation  configures  the  variables  for  a  new 
system.  This  involves  zeroing  and  recreating  the  interoperability  arrays  and  other  variables  used 
in  the  simulation.  Since  the  constants  do  not  change  from  run  to  run,  they  are  defined  up  front 
and  just  referenced  each  time. 

During  data  initialization,  the  system  generates  anywhere  from  5-20  interfaces  to  track 
using  a  uniform  distribution.  The  interface  parameters  are  stored  in  an  array  structure  and 
modified  using  MATLAB’s  array  operations  throughout  the  simulation.  Given  the  probabilities 
involved,  the  quantities  of  interfaces  for  each  system  will  alter  the  overall  OT  pass  rate.  Through 
a  series  of  test  cases,  the  authors  demonstrated  that  the  effect  of  increasing  the  number  of 
interfaces  was  minimal.  In  addition,  model  parameters  were  calibrated  based  on  the  number  of 
interfaces  to  match  literature  data,  so  if  at  a  later  point  in  time  it  is  determined  that  the  interface 
number  should  be  adjusted,  other  model  parameters  can  be  adjusted  to  maintain  model  validity. 

b.  Interoperability  Work 

The  simulation  keeps  track  of  the  interoperability  of  each  of  the  system’s  interfaces 
throughout  the  acquisition  process.  During  an  actual  implementation  on  a  real  program,  the 
details  of  the  interfaces  are  critical  to  the  evaluation  of  the  I/ORL  levels,  but  for  the  purposes  of 
the  model  those  details  are  abstracted  away.  The  model  keeps  track  of  interface  quality 
independent  of  any  details  of  the  system/subsystem  to  which  it  is  attached.  These  interface 
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interoperability  values  are  stored  in  two  separate  arrays  throughout  the  model:  one  for  the  current 
system  and  one  for  the  new  I/ORL  system. 

Prior  to  each  review,  each  of  the  critical  interfaces  had  their  interoperability  increased 
based  on  a  random  amount.  The  amount  of  interoperability  is  a  distinct,  normally-distributed 
random  variable.  Although  it  is  unique  for  each  of  the  interfaces  for  the  system,  the  amount  of 
interoperability  increase  between  technical  reviews  is  identical  between  the  current  process  and 
the  new  I/ORL  Process  (i.e.  interface  1  current  =  interface  1  new,  interface  2  current  =  interface 
2  new,  etc.).  As  mentioned  in  the  assumptions,  the  math  behind  the  model  fundamentally 
assumes  that  interoperability  is  a  number  on  an  interval  scale.  This  assumption  was  made  by  the 
authors  because  the  level  of  interoperability  was  an  abstraction  of  a  measurement  of  design 
maturity,  and  that  measurement  was  defined  to  be  on  an  interval  scale.  It  should  be  noted  that 
because  the  system  is  interval  does  not  imply  linearity  (i.e.  the  amount  of  effort,  cost,  schedule, 
etc.  to  go  from  an  interoperability  value  of  1  to  2  may  not  be  the  same  as  from  2  to  3). 

c.  I/ORL  Measurement 

Once  the  new  interoperability  values  are  generated,  the  simulation  models  the 
measurement  error  for  assigning  the  I/ORL  values.  The  process  model  used  is  a  simple 
probabilistic  model  for  assigning  correct  values,  false  positives,  and  false  negatives.  If  a  correct 
result  is  generated,  the  system  rounds  the  I/ORL  value  and  uses  that.  If  either  a  false  positive  or 
false  negative  result  is  generated,  the  I/ORL  value  is  rounded  and  the  result  is  either  added  to  or 
subtracted  from  the  actual  interoperability.  Note  that  this  process  is  distinct  for  each  of  the 
system  interfaces. 

d.  Review  Decision 

The  I/ORL  values  are  then  compared  to  the  threshold  values  for  the  technical  review  in 
question.  If  the  measured  value  for  a  given  interface  is  above  the  threshold,  it  passes  and  if  it  is 
below  it  fails  and  heads  for  rework.  The  technical  reviews  evaluated  are  ASR,  SRR,  SFR,  PDR, 
CDR,  IRR,  FRR,  and  OTRR. 

e.  Interoperability  Rework 

Each  system  interface  that  fails  a  review  has  its  interoperability  reworked  at  an  additional 
cost  proportional  to  its  developmental  status.  Since  the  cost  data  associated  with  the  failures 
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tracks  a  fully  fixed  problem,  the  amount  of  rework  done  is  assumed  to  be  exactly  equal  to  the 
work  necessary  to  pass.  In  other  words,  any  interface  that  is  reworked  at  a  technical  review  has 
an  interoperability  equivalent  to  the  threshold. 

If  a  system  is  reworked  as  described  above,  additional  cost  is  incurred  to  the  program. 
This  cost  is  review-based  and  is  detennined  by  a  table  lookup.  The  cost  is  incurred  on  a  per 
interface  basis  (i.e.  the  cost  to  fix  two  interfaces  is  twice  as  expensive  as  the  cost  to  fix  a  single 
interface)  and  does  not  factor  how  much  rework  needs  to  be  done  (i.e.  if  the  system  is  deficient 
by  .01  or  .1  interoperability  units,  the  cost  is  the  same).  Note  that  given  a  false  negative  result, 
the  system  could  be  sent  back  into  a  rework  state  (and  thus  incur  cost)  even  though  there  is  no 
problem  to  fix. 

f  Operational  Test  Evaluation 

At  this  point,  the  system  is  now  ready  to  enter  operational  test.  If  all  of  the  system's 
actual  interoperability  values  are  above  the  threshold  of  7,  the  system  passes  operational  test. 
While  the  perceived  I/ORL  value  is  used  to  determine  if  the  system  is  ready  to  enter  OT&E,  the 
system’s  actual  interoperability  is  being  tested  during  the  OT&E  phase  and  therefore  is  used  as 
the  pass  criteria.  If  a  single  interface  is  deficient,  the  system  is  considered  to  have  failed  for  the 
purposes  of  the  simulation  and  is  sent  to  the  final  rework  phase.  Similar  to  the  rework  stage,  if  a 
system  fails  operational  test,  rework  costs  are  incurred  based  on  the  number  of  interfaces  that 
failed  OT. 

g.  Data  Storage 

During  the  data  storage  phase  the  results  are  synthesized  and  stored  for  future  analysis. 
After  the  simulation  is  complete,  the  total  number  of  failures  are  tallied  and  descriptive  statistics 
for  the  current  and  new  systems  are  printed  to  the  screen.  The  statistics  of  key  importance  are 
the  average  minimum  I/ORL  value  for  each  of  the  systems,  the  OT  pass  rate,  and  the  total  cost 
overrun  which  could  be  correlated  to  schedule  slip. 

In  addition  to  the  table,  a  histogram  is  generated  summarizing  each  of  the  systems.  The 
histogram  shows  the  differences  in  minimum  I/ORL  value  for  the  current  process  and  the  one 
proposed  by  the  authors.  Figure  30  is  a  screen  capture  of  a  typical  model  result. 
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New  to  MATLAB?  Watch  this  Video,  see  Demos,  or  read  Getting  Started. 


Mo 

3/1 

3/1 

3/1 

3/1 

'1C 

'1C 

3/1 

4/1 


3 

x 

3 


ans  = 


1  (1000  Samples) 1  1  Old1 

•Min  1/ ORL 1  [6.9694] 

■%  OT  Pass1  [0.4830] 

•Cost  Overrun1  [0.3770] 


1  New1 
[7.4481] 
[0.9400] 
[0.3278] 


Elapsed  time  is  1.865530  seconds. 


Figure  30.  MATLAB  Model  Output 

This  figure  displays  the  model  output  from  MATLAB.  The  simulation  outputs  two  primary  pieces  of  data.  The  first 
is  a  table  that  displays  the  three  primary’  model  statistics:  the  average  minimum  I/ORL  value  for  the  systems,  the  OT 
pass  rate,  and  the  percentage  cost  overrun.  The  second  output  is  a  histogram  of  the  minimum  I/ORL  value  for  each 
of  the  systems  (blue)  compared  to  the  minimum  I/ORL  pass  criteria  (red). 

3.  Verification,  Validation,  and  Accreditation 

Once  the  model  structure  was  created,  the  underlying  parameters  had  to  be  gathered  such 
that  the  model  generates  valid  results.  The  three  fundamental  pieces  of  data  that  the  authors 
detennined  were  necessary  to  validate  the  model  were  interoperability  work  done  between  each 
review  phase,  costs  of  reworking  an  error,  and  the  accuracy  with  which  a  team  can  apply 
I/ORLs. 

Because  of  a  lack  of  access  to  the  necessary  data,  the  authors  ended  up  “reverse 
engineering”  a  majority  of  the  unknown  parameters.  To  accomplish  this,  the  authors  gathered 
data  on  the  high  level  SE  process  results  (for  example  OT  pass,  cost  overruns,  etc.)  and  adjusted 
the  underlying  model  parameters  until  the  model  output  matched. 
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a.  Interoperability  Work 

As  mentioned  previously,  the  simulation  tracks  an  abstract  interoperability  value  for  each 
of  the  interfaces  in  the  system.  In  between  each  of  the  reviews,  the  interoperability  of  the  system 
interfaces  increases:  this  increase  is  tenned  “interoperability  work.”  For  the  sake  of  simplicity, 
this  interoperability  work  is  defined  to  be  a  random  number  for  each  interface  at  each  review 
based  on  a  positive  nonnal  distribution  with  a  particular  mean  and  standard  deviation. 

There  were  many  particular  random  distributions  to  choose  from,  but,  as  mentioned  in  the 
assumptions,  there  were  a  series  of  characteristics  that  drove  the  decision.  The  first  was  that  the 
distribution  would  be  well  behaved.  The  authors  made  the  assumption  that  most  programs 
would  only  enter  a  milestone  review  if  all  of  their  interfaces  were  deemed  passing  or,  at  the  very 
least,  near  passing.  This  obviously  is  not  always  the  case,  but  that  variation  is  accounted  for  by 
the  randomness  in  the  distribution.  The  next  one  was  that  there  would  be  an  equal  probability  for 
the  work  to  be  above  or  below  the  mean.  The  last  factor  in  choosing  the  distribution  of  choice 
was  ease  of  programming.  The  authors  wanted  to  make  sure  that  the  simulation  could  be  run  a 
statistically  significant  number  of  times  to  provide  good  results.  Given  these  assumptions,  the 
authors  chose  a  normal  distribution.  Given  more  time  and  research,  other  alternatives  could  be 
found  that  better  fit  the  assumptions  and  literature  research  on  interoperability. 

To  detennine  the  mean  and  standard  deviation  of  the  random  interoperability  work  done, 
a  literature  search  was  conducted  but  few  direct  sources  could  be  found.  The  method  that  the 
authors  ultimately  decided  upon  is  to  manually  create  the  model  parameters  to  fit  existing  data. 
Because  the  model  compares  the  current  SE  process  to  the  new  I/ORL  system,  the  output  of  the 
current  SE  process  portion  of  the  model  should  match  data  from  existing  processes.  These 
parameters  were  then  used  for  the  I/ORL  system  as  well  to  detennine  the  relative  change. 
Leveraging  the  work  done  by  Eric  Honour  on  the  effectiveness  of  the  SE  process,  the  authors 
matched  the  model  results  to  failure  rates  found  in  large  engineering  projects  [Honour,  2004]. 

By  assuming  that  an  equivalent  amount  of  work  is  done  between  each  of  the  reviews,  the 
parameters  were  adjusted  until  the  failure  rate  for  the  current  SE  process,  as  calculated  by  the 
model,  matched  the  literature. 

The  values  that  were  ultimately  chosen  were  a  mean  of  1 . 1 1 5  and  a  standard  deviation  of 
.215.  When  these  values  are  implemented  in  the  model,  the  majority  of  the  interfaces  on  a  given 
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system  demonstrated  sufficient  design  maturity  to  pass  each  review,  but  it  is  relatively  common 
for  one  or  more  to  fail  and  have  to  be  reworked  at  each  review. 

b.  Rework  Costs 

The  most  important  parameter  for  detennining  the  cost-effectiveness  of  the  I/ORL  system 
is  the  improvement  to  the  cost  and  schedule  of  the  programs  involved.  The  theory  dictates  that 
catching  problems  early  in  the  SE  process  is  more  cost  effective  and  saves  time,  and  the  I/ORL 
Process  is  designed  to  track  problems  such  that  they  can  be  taken  care  of  early. 

The  authors  were  familiar  with  the  general  rule  of  thumb  that  as  the  system  moves 
through  the  different  phases  of  systems  engineering  (requirements  definition,  detailed  design, 
developmental  test,  operations,  etc.),  the  cost  of  finding  and  making  changes  increases  an  order 
of  magnitude  between  each  review.  The  authors  detennined  that  if  actual  data  could  be  acquired 
(rather  than  rough  approximations),  it  would  give  the  model  the  necessary  inputs  to  accurately 
calculate  the  total  nonnalized  program  cost. 

The  authors  returned  to  Eric  Honour’s  work  on  the  ROI  of  Systems  Engineering  which 
compiled  a  list  of  past  studies  to  provide  an  accurate  approximation  for  the  relative  cost 
differences  between  the  costs  of  changes  at  different  phases  of  a  software  engineering  project 
[Honour,  2007].  Unfortunately,  Honour  only  provides  relative  rankings  of  the  data  and  does  not 
provide  a  reference  for  the  absolute  cost  of  a  change,  so  it  is  only  of  limited  use  on  its  own. 

To  accommodate  this,  the  authors  developed  a  baseline  for  the  relative  data.  Rather  than 
trying  to  determine  the  absolute  cost  of  one  of  the  changes  at  a  particular  review  and 
extrapolating  for  the  remainder,  the  authors  decided  to  use  a  similar  approach  to  the  OT  pass  rate 
and  match  total  cost  overrun  for  the  model’s  output  for  the  standard  SE  process  to  actual  data. 
Upon  revisiting  the  Value  of  Systems  Engineering  work,  Honour’s  graphs  indicated  that  the  cost 
overrun  of  an  average  system  was  about  40-50%,  so  the  authors  scaled  each  of  the  relative  costs 
to  generate  a  total  average  cost  overrun  of  40-50%  [Honour,  2004]. 

The  authors  were  also  interested  in  finding  schedule  slippage  infonnation  in  addition  to 
the  cost  data.  Unfortunately,  no  schedule  slippage  data  was  available  as  a  function  of  time. 
Although  this  assumption  does  not  hold  for  all  systems,  one  can  assume  that  the  bulk  of  the  costs 
will  be  labor  and  thus  the  average  schedule  slippage  savings  would  be  approximately 
proportional  to  cost  savings. 
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c.  I/ORL  Application  Accuracy 

With  any  system  of  evaluation,  the  accuracy  at  which  that  system  can  be  applied  is 
paramount  to  its  success  and  effectiveness.  Obviously  there  is  no  actual  data  on  I/ORLs,  so  the 
authors  began  looking  for  the  effectiveness  of  TRLs,  assuming  that  the  results  would  be 
reasonably  similar.  Unfortunately,  data  outlining  the  application  accuracy  of  TRLs  was  unable 
to  be  found.  To  tackle  this  problem,  the  authors  developed  three  potential  options: 

•  Assume  100%  accuracy 

•  Assume  a  rough  accuracy  (<100%)  and  document  the  assumption 

•  Generate  an  accuracy  model  based  on  well-founded  Human  Reliability  Analysis 

equations. 

The  first  option,  while  the  easiest,  was  not  a  viable  option  given  that,  without 
measurement  inaccuracy,  systems  would  never  fail  OT&E.  In  other  words,  if  the  evaluators 
always  knew  ahead  of  time  that  a  system  would  fail  OT&E,  the  system  would  never  enter 
OT&E,  and  thus  never  fail. 

The  third  option  would  provide  the  most  validity,  but  it  would  also  be  the  most  complex 
model  to  generate.  It  would  give  the  authors  the  ability  to  compare  the  accuracy  rates  of 
experienced  versus  inexperienced  operators,  at  different  reviews,  etc.,  but  fundamentally  those 
effects  were  deemed  to  be  minor,  if  any,  when  the  simulation  was  run  for  upwards  of  10,000  data 
points. 

Therefore  the  second  option  was  the  only  one  that  was  feasible  given  the  project’s 
constraints.  The  authors  were  not  comfortable  inventing  numbers  without  some  basis  behind 
them,  so  a  literature  search  was  conducted.  No  papers  were  found  for  the  effectiveness  of 
technical  evaluation  methodologies,  but  there  is  a  broad  and  extensive  field  studying  the 
accuracy  of  medical  diagnoses.  While  this  was  not  generally  applicable  for  the  application  in 
question,  it  did  give  validity  that  the  numbers  chosen  were  accurate  within  an  order  of  magnitude 
[Bagnato,  2004], 

Ultimately  the  accuracy  of  the  measurement  was  defined  to  be  80%  accuracy  with  a  false 
positive  rate  of  15%  and  a  false  negative  rate  of  5%.  These  numbers  do  not  change  the  ultimate 
validity  of  the  model  and,  as  more  data  becomes  available,  the  analysis  can  be  modified  to 
incorporate  more  accurate  findings. 
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4.  Model  Results  and  Conclusions 


The  model  was  run  for  100,000  cycles  and  the  following  results  were  generated. 


Table  17.  I/ORL  Model  Statistics 


(100,000  Samples) 

Current  SE  Process 

I/ORL  Process 

Minimum  I/ORL 

6.9554 

7.4565 

%  OT  Pass 

46.13% 

93.73% 

Rework  Costs 

44.26% 

37.97% 

The  minimum  I/ORL  values  is  the  lowest  I/ORL  value  of  all  of  the  systems  ’  interfaces  at  OT  averaged  over  all  of  the 
number  of  runs;  the  %  OT  pass  is  the  percentage  of  the  time  that  the  systems  had  all  of  the  interfaces  above  the  7 
I/ORL  limit  for  passing  OT;  and  rework  costs  is  the  average  of  the  total  amount  of  additional  costs  generated 

because  of  failures  to  meet  the  required  thresholds. 

Table  17  indicates  two  major  facts  about  the  model.  The  first  is  that  the  model  results  for 
the  current  systems  engineering  process  matches  data  gathered  from  real  systems  in  the  actual 
DoD  acquisition  process.  This  gives  the  authors  significant  confidence  that  the  model  itself  is 
valid. 

Given  that  the  first  point  indicates  that  the  model  is  valid,  the  second  point  is  that  the 
proposed  process  is  significantly  better  than  the  current  SE  process  at  developing  systems  that 
pass  OT  on  the  first  try.  Assuming  that  the  technical  review  decisions  are  abided  by,  systems  in 
the  new  process  will  pass  more  than  90%  of  the  time  on  the  first  OT  attempt.  The  system  should 
also  reduce  the  total  cost  overrun  for  the  programs  implementing  I/ORLs  by  an  average  of  6.29% 
of  program  cost.  Although  these  cost  savings  seem  minor,  there  is  the  potential  to  save  millions 
of  dollars  across  DOD  in  the  course  of  a  fiscal  year.  Additionally  the  model  does  show  that 
I/ORLs  will  improve  the  chances  that  systems  will  meet  their  interoperability  goals  the  first  time. 

5.  Model  Areas  of  Improvement 

Although  the  authors  deemed  the  model  sufficient  to  make  a  decision  on  the  overall 
effectiveness  of  the  I/ORL  Process,  there  are  many  areas  in  which  the  simulation  can  be 
improved. 
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a.  Programming  Efficiency 


MATLAB  was  the  programming  language  of  choice  due  to  the  experience  of  the  authors 
and  the  ease  of  use.  MATLAB  is  an  interpreted  scripting  language  and  by  default  that  is  less 
efficient  than  compiled  languages.  Moving  the  model  to  something  less  processor  intensive  will 
allow  for  a  greater  number  of  data  samples  and  thus  better  statistical  accuracy. 

In  addition,  increased  efficiency  would  allow  for  greater  model  complexity  without 
sacrificing  the  required  statistical  accuracy.  This  would  allow  the  model  to  make  fewer 
assumptions  and  therefore  provide  a  more  accurate  understanding  of  the  true  effects. 

b.  Improved  Data  Sources 

As  mentioned  in  the  previous  section,  the  authors  had  difficulties  finding  data  to  exactly  match 
the  model  structure.  If  improved  sources  can  be  found,  the  model’s  code  found  in  Appendix  E 
can  be  refined  to  provide  more  accurate  results. 

c.  Study  Effects  of  Program  Size 

As  discussed  in  the  section  on  model  assumptions,  the  amount  of  interoperability  work 
done  between  technical  reviews  was  assumed  to  be  a  nonnal  distribution  with  a  fixed  mean  and 
standard  deviation.  Although  the  model  was  validated  based  on  a  nonnal  distribution,  there  may 
be  a  better  distribution  that  would  provide  more  realistic  results. 

By  setting  the  number  of  interfaces,  the  program  can  simulate  a  large  or  small  program 
moving  through  both  the  current  SE  and  modified  I/ORL  Processes.  The  authors  feel  that  an 
interesting  avenue  of  study  for  future  work  may  be  to  analyze  the  effects  of  program  size  on  the 
model’s  outputs  for  OT  pass  rate  and  rework  costs.  I/ORL  methods  may  work  well  for  large 
engineering  programs,  but  their  effects  may  not  be  as  cost  effective  on  smaller  programs  with 
fewer  interfaces  to  track. 

d.  Explore  other  distributions  for  interoperability  work 

The  authors  made  a  simplifying  assumption  that  the  interoperability  work  done  to  the  system 
interfaces  could  be  modeled  by  a  normal  distribution.  There  are  many  other  distributions  that 


117 


could  have  been  chosen  instead,  and  choosing  a  better  distribution  may  be  able  to  improve  the 
quality  of  the  results. 
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VI.  CONCLUSIONS  AND  RECOMMENDATIONS 


A.  CONCLUSIONS 

This  project  originated  with  a  need  to  detennine  why  systems  are  failing  OT.  Mission 
criticality  was  a  recurring  topic  throughout  the  preliminary  research.  The  sponsor  had  other 
groups  studying  various  aspects  of  mission  criticality;  the  authors  were  specifically  asked  to 
detennine  if  OT  failures  could  be  reduced,  through  the  application  of  mission  critical  threads. 
This  project  was  not  a  typical  SE  design  process  as  there  is  no  physical  hardware  or  software  to 
design;  rather,  a  Systems  Engineering  approach  was  applied  directly  to  improving  the  Systems 
Engineering  process. 

The  project  was  focused  and  narrowed  as  it  progressed  from  one  stage  to  the  next.  There 
was  not  sufficient  time  or  resources  for  the  Capstone  Team  to  address  every  cause  of  OT  failure. 
The  data  available  for  systems  failing  OT  indicated  that  every  system  that  failed,  did  so  for 
multiple  reasons,  the  most  prevalent  being  interoperability,  followed  by  reliability  and  training. 
Interoperability  was  prevalent  in  each  of  the  OT  failures,  with  concurrence  of  the  Sponsor  and 
the  Capstone  Advisor;  interoperability  became  the  sole  focus  of  the  new  process’s  detailed 
design.  Many  processes  were  proposed  by  the  authors,  but  ultimately  the  implementation  of 
I/ORLs  was  determined  to  be  the  most  appropriate  method  of  addressing  interoperability  issues 
in  mission  critical  threads.  An  I/ORL  rating  and  application  process  was  developed  by  the 
authors;  it  was  then  validated  and  improvements  made  through  computer  based  modeling,  as 
well  as  a  pair  of  table  top  exercises. 

Based  on  the  modeling  and  the  table  top  exercise,  the  authors  believe  that  the  current 
DoD  Systems  Engineering  process  could  benefit  from  the  implementation  of  I/ORLs.  The 
modeling  estimates  a  savings  of  approximately  6.29%  over  the  current  System  Engineering 
process.  This  seems  like  a  reasonable  result  for  implementing  I/ORLs.  Further,  the  new  process 
does  not  eliminate  the  need  for  rework;  rather  it  identifies  deficiencies  earlier  in  the  design 
process  so  they  can  be  corrected  before  further  development  occurs.  Additionally,  the  proposed 
process  has  the  potential  to  reduce  the  amount  of  failures  during  OT&E  (often  highly  visible) 
and  allow  for  earlier  assessments  and  adjustments  to  programs.  Such  adjustments  might  be 


119 


reorganization,  reevaluation  of  requirements,  increased  funding,  and  /  or  cancellation  of  a  failing 
program. 

Expectedly  the  hardest  aspect  of  the  process  to  model  is  the  human  factors.  Factors  such 
as  a  lack  of  or  varied  understanding  of  the  processes  and  definitions,  personal  biases  and 
opinions,  political  agendas,  and  human  error  are  just  some  of  the  aspects  that  are  difficult  to 
model.  These  human  factors  were  considered  and  assumptions  were  included  in  the  computer- 
based  model,  as  discussed  in  the  EORL  Application  Accuracy  Section  of  Chapter  5.  The  human 
factors  were  more  clearly  evident  during  the  Table  Top  Exercises.  Valuable  data  was  collected 
for  further  refinements  to  potentially  improve  both  the  current  process,  as  well  as  the  proposed 
addition  of  the  I/ORL  process.  The  complexity  of  the  SE  process,  along  with  the  volume  of 
sources  (government,  academia,  and  commercial)  providing  guidance,  interpretation  and 
instruction,  leads  to  variations  in  the  human  understanding  of  the  SE  process.  The  authors 
concluded  that  a  simplified  SE  process,  with  clear  concise  and  standardized  definitions  would 
help  to  improve  the  process,  with  or  without  the  implementation  of  EORLs. 

All  of  the  research  and  new  process  design  and  validation  were  then  used  to  answer  the 
Research  Questions  posed  at  the  beginning  of  the  project  development: 

Research  Question  1.  How  are  mission  critical  elements  identified  and  managed  using 
the  existing  acquisition  and  SE  processes? 

Mission  critical  elements  are  not  currently  being  identified  in  the  Systems  Engineering 
process,  and  allocations  or  dependency  on  supporting  resources  (systems,  sensors,  data,  etc.)  are 
not  being  reflected  in  the  contractor  development  plans,  and  testing  plans.  To  some  extent  they 
can  be  found  in  a  more  general  fonn  as  KPPs,  or  COIs.  However,  not  all  of  these  are  critical  to 
perfonn  a  given  mission  and  the  mission  requirements  may  need  to  be  more  specific. 

Research  Question2.  What  are  the  common  failures  in  the  engineering  process  that 
result  in  missing  mission  critical  elements?  Where  do  these  failures  occur? 

The  authors  conducted  research  on  both  the  system  engineering  process  and  on  systems 
that  experienced  difficulties  during  operational  test  and  evaluation.  This  research  indicates  that 
significant  emphasis  is  placed  on  the  Defense  Acquisition  System  measurable  perfonnance 
parameters.  The  result  is  an  over  emphasis  of  measurable  parameters,  which  in  some  cases, 
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results  in  a  loss  of  the  intent  of  the  requirement,  thus  resulting  in  complications  later  in 
development.  A  lack  of  communication  between  the  development  community,  the  user 
community,  and  the  operational  test  and  evaluation  community  was  noted.  This  may  result  in  a 
duplication  of  effort  with  regard  to  testing  the  systems,  but  more  importantly  because  of  the  lack 
of  communication,  the  testing  conducted  varies  between  DT  and  OT,  between  different  test 
agencies  perfonning  the  same/similar  testing  and  from  one  system  to  the  next.  This  means  that  a 
system  is  evaluated  according  to  one  interpretation  of  the  requirements  during  developmental 
testing  (DT)  and  another  during  Operational  Test  and  Evaluation  (OT&E).  These  failures  were 
observed  in  the  form  of  problems  with  interoperability,  reliability,  training,  suitability,  and 
effectiveness.  For  the  purposes  of  this  report,  the  authors  have  chosen  to  address  the  issue  of 
systems  passing  DT  but  failing  OT&E  as  a  result  of  a  lack  of  interoperability.  This  is  a  mission 
critical  area;  however,  when  compared  to  standard  performance  goals  it  can  be  nebulous  and 
difficult  to  test  or  simulate  in  a  realistic  way. 

Research  Question  3.  How  prevalent  are  mission  critical  failures  of programs 
discovered  during  OT&E? 

Mission  critical  failures  are  a  serious  problem.  Systems  are  failing  to  meet  their 
operational  effectiveness  or  suitability  requirements  at  a  rate  of  approximately  50%  [Defense 
Science  Board  Task  Force,  2008].  The  high  rate  of  failures  and  the  cost  associated  with 
reworking  these  systems  necessitates  that  an  approach  to  track  and  monitor  mission  critical 
elements  be  identified.  While  the  failure  rate  included  much  more  then  interoperability,  the 
authors  did  find  that  interoperability  was  the  most  prevalent.  Additionally,  a  secondary  effect  of 
fixing  interoperability  issues  earlier  in  the  design  and  development  process  is  that  it  will  allow 
for  resources  to  be  re-allocated  and  cost  savings  to  be  applied  to  other  critical  areas  of  system. 

Research  Question  4.  What  kinds  of  Modeling  and  Simulation  (M&S)  can  be  leveraged 
to  analyze  DoN  System  Engineering  processes? 

There  are  many  M&S  techniques  available.  For  this  project  we  found  two  techniques 
useful;  both  a  Table  Top  Exercise  and  statistical/mathematical  analysis  through  software 
simulation.  Table  Top  Exercises  are  a  good  method  for  evaluating  single  steps  within  the  DODI 
5000.02  process;  these  exercises  allowed  us  to  focus  on  deficiencies  in  both  the  current  process 


121 


as  well  as  the  recommended  improved  process.  In  addition  to  finding  flaws  in  the  systems,  the 
Table  Top  Exercises  allow  for  the  intentional  introduction  of  specific  events,  challenges,  missing 
or  incorrect  data  and  allow  the  observers  to  see  how  the  process  proceeds  when  the  human 
element  is  included.  Unfortunately,  running  a  table  top  exercise  to  simulate  the  whole  of  the 
DODI  5000.02  requirements  process  is  too  costly  and  time  consuming.  The  cost  and  time  issues 
significantly  outweigh  any  benefit  from  conducting  such  a  large  exercise.  For  this  reason,  the 
entire  process  was  simulated  as  a  mathematical  process.  While  numerous  modeling  and 
simulation  software  programs  are  capable,  MATLAB  was  chosen  based  on  the  skill  sets  of  the 
Research  Project  authors.  MATLAB  was  used  to  simulate  the  higher  level  steps  of  the  overall 
process,  both  current  and  proposed.  The  analysis  included  a  comparison  of  estimated  costs  for 
both  processes,  as  well  as  assessing  the  quality  of  the  interoperability  readiness  of  the  simulated 
products  for  both  processes.  To  achieve  this,  EORLs  were  assigned  to  the  simulated  products  in 
accordance  with  the  proposed  process.  For  comparison  the  simulation  evaluated  the  output  of 
current  process  and  assigned  an  I/ORL  after  the  current  process  was  complete.  These  two 
I/ORLs  were  used  to  compare  the  two  processes. 

Research  Question  5.  Can  process  improvements  be  used  to  supplement  the  DoN  System 
Engineering  processes  to  improve  handling  of  mission  critical  elements  of programs? 

The  authors  developed  a  process  to  assess  the  interoperability  maturity  of  a  system.  To 
accomplish  this,  the  authors  developed  FORLs  as  a  tool  to  assess  the  maturity  of  interoperability 
throughout  the  design  process  of  a  system.  This  improvement  evaluates  the  developmental  status 
of  capabilities  and  interfaces  that  allow  the  system  to  interoperate  with  other  legacy  and  new 
systems  in  support  of  mission  critical  threads.  I/ORL  values  are  assigned  throughout  the  process 
and  are  used  to  predict  the  system’s  ability  to  pass  operational  test  and  evaluation  with  regard  to 
interoperability.  This  report  outlines  the  proposed  process  and  provides  a  list  of  possible 
improvements. 

Research  Question  6.  What  are  the  cost  ramifications  and  possible  benefits  of 
implementing  these  enhanced  processes? 

As  demonstrated  by  the  modeling,  by  identifying  problems  earlier,  these  enhanced 
processes  have  the  potential  to  decrease  developmental  costs  by  6.29%.  The  earlier  problems  are 
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addressed,  the  less  rework  is  required  in  both  time  and  money.  Equally  importantly,  the  model 
shows  an  increased  OT&E  pass  rate  through  an  increase  in  systems  engineering  efficiency; 
leading  to  decreasing  costs  (both  time  and  money). 


B.  RECOMMENDATION 

It  is  the  recommendation  of  the  authors  that  DOD  included  the  I/ORL  proposed 
process  as  part  of  an  improved  SETR  process.  While  introducing  additional  EORL  requirements 
and  simultaneously  streamlining  the  process  may  seem  contradictory,  they  are  mutually 
beneficial.  The  streamlining  suggested  is  a  more  concise,  articulated  set  of  rules  and  definitions 
for  program  managers,  systems  engineers,  contractors,  users,  and  financial  personnel.  This 
streamlining  includes  providing  clear  definitions  and  guidance  on  the  use  of  mission  criticality. 
The  current  process  is  improved  upon  by  introducing  I/ORL’s.  However,  the  integration  of  the 
human  into  the  SE  process  still  requires  improvement,  as  is  expected  in  any  process  where 
humans  are  involved.  The  additional  information  provided  to  programs  in  the  form  of  I/ORL 
requirements  and  clarifying  definitions  will  help  to  improved  interoperability  in  system 
development. 


C.  AREAS  FOR  FURTHER  STUDY 

Throughout  the  process,  the  authors  realized  there  were  numerous  areas  that  could  not  be 
studied  by  the  authors  do  to  a  lack  of  resources  and  time.  Early  on  in  the  project,  it  was  realized 
that  the  scope  of  the  project  needed  to  be  reduced  to  a  more  manageable  size.  The  initial 
research  pointed  to  three  common  causes  of  OT  failure:  interoperability,  reliability,  and  training 
issues.  The  authors  chose  to  study  interoperability,  as  every  system  studied  in  the  research  failed 
OT  with  some  interoperability  issues,  while  only  75%  failed  with  reliability  and  training  issues. 
No  further  investigation  was  made  into  the  cause  of  reliability  or  training  failures  after  the  down 
scoping  of  the  project.  These  have  the  potential  to  become  separate  projects  in  and  of 
themselves. 

Another  area  that  was  briefly  discussed  during  the  down  select  is  the  cost  of 
implementing  the  new  process.  The  authors  are  aware  there  are  costs  associated  with  training, 
implementing,  and  maintaining  process;  no  analysis  was  done  to  estimate  the  actual  cost  of 
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implementing  the  new  system  or  the  administrative  burden.  Despite  any  possible  increased  costs 
during  initial  implementation  it  is  assumed  during  the  down  selection  that: 

•  All  of  the  new  processes  considered  have  training  and  implementation  costs. 

•  The  long  term  savings  of  implementing  a  new  process  will  far  outweigh  the  cost  of 
process  implementation. 

•  The  new  process  would  result  in  better  OT  results  than  the  current  process. 

It  is  suggested  that  an  additional  study  be  done  to  detennine  the  actual  cost  of 
implementation  and  training  on  the  new  processes.  It  is  likely  that  these  costs  can  be  mitigated 
to  some  degree  by  combining  with  other  proposed  changes  to  the  SE  process;  however,  the 
authors  did  not  conduct  research  to  determine  the  cost  or  timeline  to  implement  such  process 
improvements. 
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I.  Introduction 

This  is  the  Capstone  Project  Management  Plan  (PMP)  for  the  Naval  Sea  Systems  Command 
(NAVSEA)  Cohort  3 1 1-092S,  hereto  after  referred  to  as  the  Mission  Critical  Team.  As  part  of 
the  Naval  Postgraduate  School  (NPS)  Master  of  Science  in  System  Engineering  (MSSE) 
Capstone  Project,  the  Mission  Critical  Team  will  investigate  changes  to  the  requirements 
identification  and  management  processes  that  a  system  will  utilize  during  the  Department  of  the 
Navy  System  Engineering  process. 

1.1  Problem  Statement 

The  current  Department  of  the  Navy  (DoN)  system  engineering  process  has  documented 
instances  of  programs  failing  to  detect  critical  inter-operational  failures  prior  to  operational  level 
testing.  Programs  are  successfully  passing  developmental  testing;  however,  these  tests  do  not 
provide  adequate  assurances  that  the  system  will  satisfy  user  needs  or  CONOPS-based 
interoperating  requirements  of  the  operational  community.  Furthermore,  requirements  that  are 
critical  to  mission  success  are  not  being  identified  in  the  current  SE  process,  and  allocations  or 
dependency  on  supporting  resources  (systems,  sensors,  data,  etc.)  are  not  being  reflected  in  the 
system  engineering,  contractor  development  plans,  and  testing  plans. 

DoDI  5000.02  and  related  documents  describe  the  Defense  Acquisition  System  used  for 
development  of  Department  of  the  Navy  programs.  Despite  extensive  guidance  and  multiple 
decision  gates,  programs  are  still  failing  to  ensure  that  all  requirements  critical  to  meeting  the 
user  need  are  identified  and  properly  tested  prior  to  the  start  of  the  Operational  Test  and 
Evaluation  (OT&E)  process.  Improvements  must  also  be  made  to  the  system  engineering 
requirements  process  to  ensure  that  these  critical  requirements  are  made  visible  at  key  technical 
reviews(s)  such  as  Preliminary  Design  Review  (PDR),  Critical  Design  Review  (CDR),  Test 
Readiness  Review  (TRR),  and  Production  Readiness  Review  (PRR). 

1.2  Research  Questions 

The  following  research  questions  will  be  answered  as  the  project  progresses: 

>  How  are  mission  critical  elements  identified  and  managed  using  the  existing  acquisition 
and  SE  processes? 

>  What  are  the  common  failures  in  the  engineering  process  that  result  in  missing  mission 
critical  elements?  Where  do  these  failures  occur? 

>  How  prevalent  are  mission  critical  failures  of  programs  discovered  during  OT&E? 
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>  What  kinds  of  Modeling  and  Simulation  (M&S)  can  be  leveraged  to  analyze  DoDI 

5000.02  processes? 

>  Can  process  improvements  be  used  to  supplement  the  DoDI  5000.02  processes  to 

improve  handling  of  mission  critical  elements  of  programs? 

>  What  are  the  cost  ramifications  and  possible  benefits  of  implementing  these  enhanced 

processes? 

1.3  Stakeholders 

>  Naval  War  Fighter 

The  key  stakeholder  for  this  project  is  the  naval  war  fighter.  Although  the  war  fighter 
does  not  encounter  the  identified  problem  until  a  system  becomes  operational,  the  war 
fighter  is  left  to  deal  with  consequences  of  an  ineffective  system  that  is  delivered  with 
deficiencies.  The  Joint  Capabilities  Integration  and  Development  System  (JCIDS) 
process  describes  the  Joint  Forces  Command  (JFCOM)  as  the  war  fighter  representative 
into  the  acquisition  infrastructure.  Commander,  US  Fleet  Forces  Command  (CFFC)  is  the 
approval  authority  for  the  Capability  Development  Documents  (CDDs)  and  Capability 
Production  Documents  (CPDs). 

>  ASN  (RDA)  CHENG 

The  Assistant  Secretary  of  the  Navy  for  Research,  Development  and  Acquisition  Chief 
Engineer’s  Office  is  responsible  for  System  Engineering  Technical  Reviews  (SETR)  for 
all  ACAT  programs,  regardless  of  the  ACAT  designation  ASN  (RDA)  CHENG  will  be 
interested  in  the  outcome  of  this  project  as  our  sponsor.  We  will  be  engaged  with  their 
representative  on  a  continuous  basis  to  understand  and  receive  guidance  on  the  direction 
of  the  project.  They  will  also  be  interested  in  the  utilization  of  the  possible  process 
changes  for  incorporation  into  the  SETR  process. 

>  Program  Office/Program  Managers 

The  Program  Offices  and  Program  Managers  for  specific  development  systems  may  be 
impacted  by  this  project.  The  programs/systems  may  be  able  to  utilize  the  output  of  this 
project  to  re-define  their  systems  engineering  process,  requirements  definition,  test 
planning  and  conduct. 
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>  System  Design,  Development  and  Validation  Teams 

The  design  engineering  teams  may  be  affected  by  the  outcome  of  this  project.  This  could 
be  the  prime  contractors  or  the  SYSCOM  field  activities  charged  with  the  system 
development  efforts.  The  criteria  identified  for  “mission  criticality”  may  have  impacts  on 
the  design  and  development  of  a  system,  along  with  the  testing  requirements  identified. 

>  U.S.  Navy  Operational  Test  and  Evaluation  Force  (OPTEVFOR) 

OPTEVFOR  assesses  the  operational  effectiveness  and  suitability  of  new  and  improved 
war  fighting  systems  and  capabilities.  The  outcome  of  this  project  may  provide 
additional  information  in  order  to  evaluate  systems  under  test. 

1.4  Project  Proposal 

Based  on  the  need  to  address  this  deficiency  in  the  current  system  development  process, 
the  current  DoDI  5000.02  process  will  be  analyzed  by  investigating  methods  to  improve 
capturing,  monitoring,  implementing  and  testing  the  requirements  critical  to  ensuring  mission 
success.  Improvements  to  the  DoDI  5000.02  processes  will  be  developed  to  ensure  that  these 
requirements  are  identified  and  promulgated  through  key  program  documents  and  design 
reviews.  The  objective  of  this  capstone  project  will  be  to  develop  enhanced  processes 
supplementing  DoDI  5000.02  to  improve  the  handling  of  these  requirements.  These  enhanced 
processes  will  be  simulated  on  a  DoD  system  in  a  tabletop  exercise  to  provide  an  example  of 
how  this  modified  process  should  be  performed.  The  enhanced  process  and  the  demonstration 
will  be  documented  in  the  capstone  final  report. 

System  engineering  principles  and  methodologies  will  be  used  to  propose  changes  to  the 
requirements  processes  in  the  DoN  acquisition  system  to  ensure  that  requirements  that  are 
critical  to  mission  success  are  identified  and  ensure  that  these  requirements  are  integrated  into 
the  system  development  plans  and  testing  plans.  These  requirements  must  be  aligned  with  the 
user  needs  and  system  requirements  and  must  be  testable  at  the  system  and  interoperability 
levels.  This  process  is  initiated  at  the  system  needs  analysis  phase  and  continues  through  the 
operational  testing  phase.  Development  of  these  requirements  requires  the  early  involvement 
and  support  of  the  users  and  testers  in  order  to  ensure  that  operational  testing  accurately 
represents  the  needs  of  the  users  and  to  align  the  system  development  process  with  the  user 
needs.  In  addition  to  ensuring  that  these  requirements  are  included  in  the  original  requirements 
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baseline,  this  process  will  ensure  that  these  requirements  are  included  in  the  appropriate 
development  and  testing  plans  and  all  applicable  design  reviews. 

A  DoN  program  will  be  selected  to  use  as  a  demonstration  of  the  enhanced  process.  This 
program  will  be  selected  based  upon  availability  of  critical  information  and  the  knowledge, 
interest,  and  experience  of  the  capstone  team.  The  critical  information  necessary  will  be  system 
requirements  as  tested  during  operational  testing  and  the  history  of  how  the  requirements  were 
detennined  from  the  user  needs  and  how  those  requirements  changed  over  the  development 
cycle.  This  demonstration  will  entail  perfonning  the  key  steps  of  the  requirements  process  as  a 
tabletop  exercise.  The  results  from  these  steps  will  be  documented  in  the  final  capstone  report. 
In  addition,  the  team  will  perform  a  cost/benefits  analysis  of  the  enhanced  process,  including  an 
assessment  of  how  the  requirements  and/or  testing  of  the  program  would  have  changed  if  this 
modified  requirements  process  had  been  performed. 

1.5  Organization 

The  Mission  Criticality  team  is  made  up  of  15  members  located  in  three  different 
geographical  locations:  NSWC  Indian  Head  MD,  NSWC  Dam  Neck  VA,  and  NSWC  Port 
Hueneme  CA.  The  cohort’s  project  orientation  is  functionally  organized  between  Project 
Management  and  Systems  Engineering,  as  indicated  in  Figure  1 .  The  roles  and  responsibilities 
of  each  team  member  may  be  re-assigned  to  meet  the  project  needs  and  requirements. 
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Figure  1  Organizational  Chart 

1.5.1  Roles  and  Responsibilities 

Table  1.  Roles  and  Responsibilities 


Role 

Responsibilities 

Advisor(s) 

Instructors(s) 

Stakeholder(s) 

♦  Provide  guidance  throughout  the  project 

Program  Manager 

♦  Interface  with  the  sponsor,  stakeholders,  instructor, 
and  advisor 

♦  To  monitor  project  progress  and  ensure  schedule 
and  documentation  requirements  are  met 

Editor  (Project  Control) 

♦  Edit  and  compile  all  documentation 

♦  Ensure  format  structure  is  maintained  throughout 
all  documentation 

Librarian/W  ebmaster(s) 

♦  Maintains  copies  of  all  submitted  drafts  and  final 
documentation  submitted  to  the  instructor  and 
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(Project  Control) 

stakeholders  (through  SAKAI) 

♦  Maintains  organization  of  SAKAI  group  webpage 

♦  Takes  minutes  for  all  meetings  involving  the  entire 
Capstone  Team 

♦  Collects  and  maintains  meeting  minutes  from  the 
leads  of  each  small  group 

Scheduler  (Project  Control) 

♦  Responsible  for  developing  and  updating  the 
schedule  and  tracking  all  due  dates  for  the  project 

Systems  Engineering  Team 

♦  All  group  members  will  participate  in  the  System 
Engineering  Process. 

♦  Ensure  the  appropriate  Systems  Engineering 
approach  is  utilized  throughout  the  project 

♦  Perfonn  all  system  engineering  tasks  as  required 

♦  Break  into  smaller  teams  to  perfonn  parallel  system 
engineering  tasks  as  appropriate 

1.5.1  Team  Member  Assignments 

The  initial  team  member  roles  are  listed  in  Table  2.  As  previously  stated,  most  members 
are  a  part  of  the  systems  engineering  team.  Their  specific  assignments  will  change  as  the  project 
progresses.  Several  systems  engineering  team  members  have  additional  aptitude  in  specific  roles 
(e.g.  modeling  and  simulation);  these  will  be  taken  into  account  during  task  assignment 


Table  2.  Team  Member  Roles  and  Responsibilities 


Name 

Initial  Roles  &  Responsibilities 

Kyle  Foley 

Co-Program  Manager 

Michael  Thrift 

Co -Pro  gram  Manager 

Eric  Hawley 

Lead  Editor 

Janet  Holt 

Librarian/W  ebmaster 

Alex  Guerao 

Scheduling  Lead 

Peggy  Rogers 

Systems  Engineering  (Backup  Editor) 

Chirana  Pimsam 

Systems  Engineering  (Backup  Librarian/Web) 

Tien  Phan 

Systems  Engineering  (Backup  Scheduler) 
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Steve  Tegtmeyer 

Systems  Engineering 

Steven  Possehl 

Systems  Engineering 

Jesus  Garcia 

Systems  Engineering 

Phong  Trinh 

Systems  Engineering 

John-Anthony  Gorospe 

Systems  Engineering 

Theodore  Schindler 

Systems  Engineering 

Mark  Cavolowsky 

Systems  Engineering 

1.6  Project  Advisors 

Dr.  Paul  Shebalin  is  Director  of  the  Wayne  E.  Meyer  Institute  of  Systems  Engineering  and 
a  Senior  Lecturer  of  Systems  Engineering  at  the  Naval  Postgraduate  School  in  Monterey,  CA. 

Mr.  Gregory  Miller  is  a  Lecturer  of  Systems  Engineering  at  the  Naval  Postgraduate  School  in 
Monterey,  CA.  These  two  professors  will  be  our  project  advisors  for  the  duration  of  the  capstone 
project. 

1.7  Risk  Management 

Risks  affecting  the  success  of  the  development  of  an  improved  process  will  be  identified 
throughout  the  project.  In  identifying  the  risks,  the  probability  of  occurrence  and  the  potential 
consequence  will  be  quantified  by  initiator  of  the  risk,  on  a  scale  from  0  to  1 .  A  risk  rating  will 
be  given  (probability  of  occurrence  times  potential  consequence),  which  will  be  used  to  prioritize 
the  risks.  Once  the  risks  are  properly  identified  and  prioritized,  mitigation  strategies  and 
contingency  plans  can  be  developed  in  order  to  reduce  the  risk  rating.  These  strategies  and  plans 
will  be  brought  to  the  team  as  a  whole  for  approval  before  taking  action  on  the  risks.  Once  these 
mitigation  strategies  and/or  contingency  plans  are  approved,  the  risk  will  be  monitored  as  the 
mitigation  steps  occur,  up  until  the  risk  will  be  at  a  point  that  the  program  is  willing  to  assume. 
The  risk  matrix  shown  in  Figure  2  will  be  used  to  visually  display  the  risk. 
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POTENTIAL  CONSEQUENCE 


Figure  2.  Risk  Matrix 

1.7  Risk  Management 

Below  are  the  initial  risks  to  this  project  as  identified  by  the  project  team.  As  the  project 
progresses,  additional  risks  will  be  added  as  they  are  identified. 

1.8.1  Personnel  Risk 

Description:  If  the  project  is  not  properly  scoped  to  the  amount  of  personnel  on  the  team,  then 
the  project  schedule  may  slip  to  the  right,  missing  the  due  date. 

>  Risk  Rating:  0.48 

■  Probability  of  Occurrence:  0.6 

■  The  problem  statement  for  the  team  has  been  established,  and  the  project 
management  plan  contains  an  attainable  schedule.  The  team  must  meet  the 
deliverables  and  milestones  in  order  to  stay  on  schedule  to  meet  the  due  date. 
The  probability  will  reduce  as  the  project  progresses  according  to  schedule. 

■  Potential  Consequence:  0.8 
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■  If  the  team  over-estimates  what  can  be  accomplished,  then  it  can  affect  the 
successful  completion  of  this  project  on  time.  This  can  result  in  having  an 
uncompleted  project  at  the  due  date. 

>  Mitigation  Steps: 

■  Keep  the  team  engaged  by  scheduling  regular  meetings. 

■  Ensure  scope  of  the  project  does  not  increase  to  the  point  where  the  team  cannot 
realistically  accomplish  the  goals. 

■  Properly  decide  when  to  move  onto  the  next  phase  of  the  project. 

1.8.2  Resource  Risk 

Description:  If  the  project  lacks  the  resources  to  collect  data  from  past  and  current  systems,  then 
the  proposed  process  (deliverable)  may  be  inadequate. 

>  Risk  Rating:  0.35 

■  Probability  of  Occurrence:  0.7 

■  The  lack  of  data  for  this  project  is  potentially  high  due  to  the  difficulty  of 
obtaining  historical  information  from  past  and  current  ACAT  acquisition 
programs.  As  the  project  progresses  and  more  data  are  received,  this  rating 
can  be  reduced. 

■  Potential  Consequence:  0.5 

■  While  this  will  unlikely  prevent  the  completion  of  the  project,  the  amount  of 
data  is  directly  related  to  the  quality  of  the  project.  The  more  data  that  can  be 
studied,  the  better  the  quality  of  the  project. 

>  Mitigation  Steps: 

■  Talk  to  various  points  of  contacts  to  obtain  historical  data  on  ACAT  Acquisition 
programs. 

■  Analyze  data  to  ensure  its  quality. 

.8.3  Changing  Requirements/Scope 

Description:  If  the  requirements  for  the  project  continuously  change/grow,  then  the  project 
schedule  may  slip  to  the  right,  missing  the  due  date. 

>  Risk  Rating:  0.64 

■  Probability  of  Occurrence:  0.8 
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■  The  project  is  currently  still  being  defined  through  the  PMP.  Requirements 
for  the  project  have  not  been  written  down.  Probability  of  occurrence  can  be 
reduced  throughout  as  the  project  progresses. 

■  Potential  Consequence:  0.8 

■  The  consequence  of  allowing  requirements  to  change  while  unmanaged  can 
increase  the  scope  of  this  project,  affecting  the  successful  completion  of  this 
project  on  time.  While  a  product  can  still  be  turned  in  at  the  due  date,  the 
product  would  be  incomplete. 

>  Mitigation  Steps: 

■  Ensure  the  scope/requirements  of  the  task  are  within  the  team’s  capability. 

■  Baseline  the  requirements  to  prevent  requirements  creep. 

■  Manage  requirements  change. 

2  Systems  Engineering  Approach 

The  approach  outlined  below  provides  the  overall  systems  engineering  process  that  will  be 
used  during  this  project. 

2.1  Overview 

The  goal  of  the  project  is  to  analyze  the  deficiencies  of  the  current  systems  engineering 
process  supporting  Department  of  Navy  (DoN)  Acquisition  in  the  area  of  requirements  critical  to 
ensuring  mission  success.  This  will  likely  include  creating  an  addendum  to  the  current  systems 
engineering  process. 

2.2  Process  Phases 

The  project  will  consist  of  five  phases:  Process  Review  and  Problem  Identification,  Analysis 
of  Alternatives,  Detailed  Process  Design,  Process  Validation,  and  Present  Recommendations 
(Figure  3).  Each  phase  will  be  executed  by  taking  the  inputs  and  feedback  from  later  phases. 
Once  the  given  outputs  are  of  suitable  quality,  then  each  phase  can  be  considered  complete.  A 
detailed  description  of  each  phase  is  found  in  the  subsequent  sections. 
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QpQ  Procesp  Guidelines 


Figure  3.  SE  Process  Diagram 


2.1.1.  Process  Review  and  Problems  Identification 

Process  Description: 

All  stakeholder  needs  and  requirements  will  be  collected  and  prioritized  based  on 
pertinence  to  the  project  objectives  and  stakeholder  importance.  Conflicting  needs/requirements 
will  be  discussed  with  the  required  stakeholders  to  reach  an  agreement/compromise  before  the 
team  baselines  the  requirements. 

The  team  will  review  the  current  systems  engineering  process  contained  within  the 
Department  of  Defense  (DoD)  acquisition  process.  The  goal  will  be  to  examine  the  process’s 
overall  effectiveness  at  identifying  the  mission  critical  requirements  and  evaluating  these 
requirements  during  developmental  testing.  The  process  will  be  evaluated  to  detennine  how 
system  level  requirements  are  developed  from  user  needs.  Further  investigation  will  be 
conducted  to  examine  how  operational  tests  are  developed  based  upon  the  system  requirements 
and  the  underlying  user  needs. 

Research  will  be  done  to  identify  recently  developed  systems  that  passed  and  failed 
operational  testing.  Each  system  will  be  examined  to  determine  sources  of  success  and  causes  of 
failure.  This  will  be  accomplished  by  addressing  questions  such  as: 


138 


>  How  are  a  system’s  requirements  captured  in  a  Capability  Development 
Document  (CDD)  supported  by  (or  connected  to)  doctrinally-defined  missions? 

>  Are  appropriate  user-centric,  solution-neutral  metrics  identified? 

>  Are  a  system’s  external  interfaces  identified? 

>  Are  realistic  Concepts  of  Operation  (Conops)  used? 

>  Were  testers  and  end  users  formally  involved  in  the  Systems  Engineering 
Technical  Review  (SETR)? 

The  system’s  ability  to  meet  their  requirements  and  objectives  that  satisfy  the 
stakeholders’  needs  will  be  analyzed  to  detennine  the  mission  critical  beneficial  concepts  and 
shortfalls  for  a  system’s  engineering  process. 

Inputs: 

>  DoD  Process  Guidelines:  VCJCS  Oversight  (CJCSI  3170.01),  USD(AT&L)  Oversight 
(DoDI  5000.02),  Navy  SETR  Handbook.,  WSARA,  and  the  Navy  NR-KPP 
Implementation. 

>  Historical  Data  based  on  Existing  Systems  (Both  Passed  and  Failed  OT) 

>  Inquiry  for  Problems  (Feedback  from  Analysis  of  Alternative  Phase) 

Outputs: 

>  Agreed  Upon  Requirements  from  Stakeholders 

>  Identified  mission  critical  beneficial  concepts  and  shortfalls  based  on  historical  data 

>  Process  Map  of  existing  SE  Process 

>  List  of  assessed  systems 

2.2.2.  Analysis  of  Alternatives 

Process  Description: 

The  analysis  of  alternatives  will  be  accomplished  following  the  identification  of  the 
source  of  failures  pertaining  to  requirements  critical  to  mission  success  within  the  system 
engineering  process.  Key  decisive  criteria  will  be  established  at  this  phase  to  detennine  feasible 
alternatives.  The  application  of  decisive  criteria  will  be  employed  to  exclude  undesirable 
solutions  and  highlight  acceptable  alternatives.  Multiple  process  concepts  will  be  developed 
based  upon  these  decisive  criteria.  A  cost  analysis  will  be  developed  for  each  of  the  concepts, 
analyzing  the  cost  of  implementation  and  the  cost  of  execution.  These  concepts  will  also  be 
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rated  based  on  the  quality  of  process  improvement,  looking  at  factors  concerning  thoroughness, 
traceability,  and  simplicity.  The  resulting  concepts  will  then  be  evaluated  and  chosen  to  develop 
the  best  mission  criticality  process  based  on  cost-effectiveness  and  process  improvement  in  order 
to  prevent  the  shortfalls  identified  in  the  previous  phase. 

The  programs  identified  in  the  previous  phase  will  be  evaluated  to  detennine  what 
program  will  be  used  as  the  representative  example.  This  evaluation  will  be  done  based  on 
availability  of  critical  information,  team  knowledge  of  the  program,  and  the  representative  nature 
of  the  program.  The  representative  nature  is  defined  as  being  of  sufficient  complexity  to 
adequately  demonstrate  the  process  and  also  the  likelihood  of  the  problems  encountered  on  that 
program.  Based  on  these  evaluations,  a  single  program  will  be  selected  as  the  representative 
example. 

Inputs: 

>  Agreed  Upon  Requirements  from  Stakeholders 

>  DoD  Process  Guidelines:  VCJCS  Oversight  (CJCSI  3170.01),  USD(AT&L)  Oversight 
(DoDI  5000.02),  Navy  SETR  Handbook.,  WSARA,  and  the  Navy  NR-KPP 
Implementation. 

>  Identified  mission  critical  beneficial  concepts  and  shortfalls  based  on  historical  data 

>  List  of  assessed  systems 

>  Process  Map  of  existing  System  Engineering  (SE)  Process 

>  Mission  Criticality  Concepts  (Feedback  from  the  Detailed  Process  Design) 

Outputs: 

>  Mission  Criticality  Process  Solution 

>  Mission  Criticality  Process  Alternatives 

>  Representative  Program 

>  Inquiry  for  Problems  (Feedback  to  Process  Review  and  Problem  Identification  Phase) 

2.2.3.  Detailed  Process  Design 

Process  Description: 

The  selected  process  concept  will  be  used  to  develop  a  mission  criticality  process  that  can 
be  integrated  into  the  DoD  system  engineering  process.  The  new  process  could  possibly  include 
new  deliverables  such  as:  new  exit/entrance  criteria,  modified  documents,  new/modified 
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instructions,  and  organizational  charts  that  describe  roles  and  responsibility.  Mission  Criticality 
Process  Alternatives  will  be  used  as  reference  material  when  encountering  detail  design  issues. 
Concepts  for  improvement  can  be  taken  from  the  alternatives  or  generated  during  the  detailed 
design.  These  concepts  can  be  used  as  inputs  in  the  Analysis  of  Alternative  phase  to  create  an 
improved  process  solution.  While  designing  the  new  process,  the  mission  criticality  validation 
criteria  must  be  developed.  In  the  next  phase,  this  process  will  be  simulated  on  the 
representative  program  selected  in  the  previous  phase. 

Inputs: 

>  Mission  Criticality  Process  Solution 

>  Mission  Criticality  Process  Alternatives 

>  Agreed  Upon  Requirements  from  Stakeholders 

>  Design  Modifications  (Feedback  from  the  Process  Validation  Phase) 

Outputs: 

>  Mission  Criticality  Detailed  Process 

>  Mission  Criticality  Process  Validation  Criteria 

>  Mission  Criticality  Concepts  (Feedback  to  the  Analysis  of  Alternatives  Phase) 

2.2.4.  Process  Validation 

Process  Description: 

The  new  mission  criticality  process  will  be  evaluated  on  the  representative  system  by 
perfonning  table-top  exercises.  The  exercises  will  use  the  validation  criteria  developed  during 
the  design/modeling  phase  to  evaluate  the  process.  Cost  data  concerning  the  execution  of  the 
process  will  be  collected  while  the  process  is  validated.  The  results  from  the  evaluation  will  be 
analyzed  and  reported. 

Inputs: 

>  Mission  Criticality  Detailed  Process 

>  Mission  Criticality  Process  Validation  Criteria 

>  Representative  Program 

Outputs: 

>  Mission  Criticality  Process  Evaluation  Report 
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>  Cost  Analysis  Report 

>  Design  Modification  (Feedback  to  the  Detailed  Process  Design  Phase) 

2.2.5.  Report  Findings  and  Make  Recommendation 

Process  Description: 

Once  the  process  is  simulated  and  analyzed,  a  final  report  can  be  generated  for  the 
sponsors,  stakeholders,  and  advisors.  The  report  will  include  how  successful  the  new  process 
was  at  addressing  the  problems  encountered  on  the  representative  system  and  how  other  systems 
examined  could  have  benefited  from  this  new  process.  The  report  will  include  a  cost  analysis  for 
implementation  of  this  process  for  future  systems.  A  presentation  will  accompany  the  report  in 
order  to  present  the  findings  of  the  team. 

Inputs: 

>  Mission  Criticality  Process  Evaluation  Report 

>  Cost  Analysis  Report 

Outputs: 

>  Final  Report  and  Recommendations 

>  Brief  Out  Presentation 

3  Milestones  &  Deliverables 

Figure  4  below  provides  the  proposed  schedule  and  milestones  for  this  capstone  project.  The 
major  milestones  are  IPR  #1,  IPR  #2,  and  the  Final  Presentation. 
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3.1.  Schedule 


Task  Name 

Duration  Start 

Phase  1  -  Process  Review 

30  days  Mon  5/3/10 

Process  Review 

25  days  Mon  5/3/10 

Review  SE  Process  Documents 

2  wks  Mon  5/3/10 

Generate  Process  Map  for  Requirements 

2  wks  Mon  5/17/10 

Independent  Validation  of  Requirements  Process 

1  wk  Mon  5/31/10 

Review  Operational  Testing  Documents 

2  wks  Mon  5/3/10 

Generate  Process  Map  for  OT  Planning 

2  wks  Mon  5/17/10 

Independent  Validation  of  OT  Planning 

1  wk  Mon  5/31/10 

System  Failure  Review 

25  days  Mon  5/3/10 

Identify  Systems  with  OT /Operational  Issues 

2  wks  Mon  5/3/10 

Research  Successes/Failures  in  those  Systems 

2  wks  Mon  5/17/10 

Determine  Shortfalls  in  SE  Process 

1  wk  Mon  5/31/10 

Prepare  for  1  PR  #1 

1  wk  Mon  6/7/10 

IPR  #1 

0  days  Mon  6/14/10 

Phase  II  -  AoA 

25  days  Tue  7/6/10 

Process  Concepts 

25  days  Tue  7/6/10 

Determine  Process  Guidelines 

2  wks  Tue  7/6/10 

Determine  Decisive  Criteria 

2  wks  Tue  7/6/10 

Develop  Process  Concepts 

2  wks  Tue  7/20/10 

Evaluate  &  Downselect 

1  wk  Tue  8/3/10 

Representative  System  Selection 

25  days  Tue  7/6/10 

Identify  Sources  of  Information  on  Each  System 

2  wks  Tue  7/6/10 

Determine  T earn  Knowledge  of  Systems 

1  wk  Tue  7/20/10 

Evaluate  Representative  Nature  of  Systems 

1  wk  Tue  7/27/10 

Evaluate  &  Downselect 

1  wk  Tue  8/3/10 

Phase  III  -  Detailed  Process  Design 

30  days  Tue  8/10/10 

Process  Development 

25  days  Tue  8/10/10 

Develop  Process  Map  for  Concept 

1  wk  Tue  8/10/10 

Develop  Draft  Procedures  for  Process 

2  wks  Tue  8/17/10 

Develop  Document  Formats 

2  wks  Tue  8/31/10 

Process  Validation  Planning 

25  days  Tue  8/10/10 

Develop  Validation  Plan 

5  wks  Tue  8/10/10 

Prepare  for  IPR#2 

1  wk  Tue  9/14/10 

IPR  #2 

0  days  Mon  9/20/10 

Phase  IV  -  Process  Validation 

30  days  Mon  9/20/10 

Develop  Validation  Process 

2  wks  Mon  9/20/10 

Perform  Validation 

3  wks  Mon  10/4/10 

Analyze  Validation  Results 

1  wk  Mon  10/25/10 

Phase  V  -  Recom  mendations 

30  days  Mon  11/1/10 

Extrapolate  Results  of  Validation 

2  wks  Mon  11/1/10 

Develop  Cost  Analysis 

2  wks  Mon  11/15/10 

Prepare  Report 

6wks  Mon  11/1/10 

Final  Presentation 

0  days  Mon  12/13/10 

May  Jun  jju!  Auq  i  Sep  Oct  Nov  Dec  Jan 


<0  12/13 


Figure  4.  Schedule  and  Milestones 


3.2.  Deliverables 

Based  on  the  need  to  address  this  deficiency  in  the  current  system  development  process, 
the  current  DoDI  5000.02  process  will  be  analyzed  by  investigating  methods  to  improve 
capturing,  monitoring,  implementing  and  testing  the  requirements  critical  to  ensuring  mission 
success.  There  will  be  two  primary  deliverables  of  this  capstone  project.  First,  enhanced 
processes  will  be  developed  supplementing  DoDI  5000.02  to  improve  the  handling  of  these 
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critical  mission  requirements.  Second,  these  enhanced  processes  will  be  simulated  on  a  DoD 
system  in  a  tabletop  exercise  to  provide  an  example  of  how  this  modified  process  should  be 
performed.  The  results  from  these  steps  will  be  documented  in  the  final  capstone  report.  In 
addition,  the  team  will  perform  a  cost/benefits  analysis  of  the  enhanced  process,  including  an 
assessment  of  how  the  requirements  and/or  testing  of  the  program  would  have  changed  if  this 
modified  requirements  process  had  been  performed. 
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APPENDIX  B:  OPERATIONAL  TEST  PLANNING  PROCESS 


The  Operational  Test  Director  (OTD)  uses  several  documents  in  preparation  for  testing. 
OTDs  have  an  obligation  to  get  involved  early  as  possible  in  the  development  of  a  new  weapon 
system  including  providing  meaningful  input  to  various  foundation  documents.  The  defense 
acquisition  system  has  undergone  significant  changes  since  the  introduction  of  the  Joint 
Capabilities  Integration  Development  System  (JCIDS)  in  2003.  Since  many  programs  continue 
to  use  legacy  documents,  the  OTD  references  legacy  and  current  foundation  documents 
including: 

•  Mission  Needs  Statement  (MNS) 

•  Operational  Requirements  Document  (ORD) 

•  Analysis  of  Alternatives  (AO A) 

•  Acquisition  Program  Baseline  (APB) 

•  Acquisition  Infonnation  Assurance  (IA)  Strategy 

•  Initial  Capabilities  Document  (ICD) 

•  Capability  Development  Document  (CDD) 

•  Capability  Production  Document  (CPD) 

1.  Mission  Need  Statement  (MNS) 

DoD  Regulation  5000. 1R,  a  legacy  document,  required  DoD  components  to  document 
deficiencies  in  capabilities  and  opportunities  to  provide  new  capabilities  in  a  MNS.  These 
capabilities  are  expressed  in  broad  operational  terms  in  the  MNS.  System  performance 
objectives  and  minimum  acceptable  requirements  were  developed  from  the  MNS  as  part  of  the 
development  of  the  ORD. 

2.  Operational  Requirements  Document  (ORD) 

DoD  Regulation  5000. 1R,  a  legacy  document,  required  the  use  of  the  ORD  to  document 
system  requirements.  The  JCIDS  process  uses  capabilities  documents  for  system  definition, 
some  programs  may  still  use  an  ORD  to  define  system  requirements. 

3.  Analysis  Of  Alternatives  (AO A) 

DoD  Instruction  5000.2  describes  the  use,  fonnat,  and  content  of  an  AOA.  The  AOA  is 
an  analytical  comparison  of  operational  effectiveness,  suitability,  and  life-cycle  cost  of 
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alternatives  that  satisfy  established  mission  capability  requirements.  The  OTD  may  use  the  AOA 
as  a  source  document  supporting  MBTB  and  TEMP  development. 

4.  Acquisition  Program  Baseline  (APB) 

The  PM  initially  develops  the  APB  as  a  concept  baseline  prior  to  program  initiation.  A 
development  baseline  and  a  production  baseline  are  prepared  for  MS-B  and  MS-C  capturing  the 
key  parameters  that  define  the  system  and  listing  the  objectives  and  thresholds  are  listed.  Key 
parameters  are  the  Measures  of  Effectiveness  (MOEs)  and  the  Measures  of  Suitability  (MOSs) 
identified  in  the  requirements/capabilities  document.  The  OTD  reviews  the  APB  to  ensure 
consistency  between  the  requirements/capabilities  document,  the  baseline  which  establishes 
explicit  perfonnance  and  thresholds,  and  the  TEMP. 

5.  Acquisition  Information  Assurance  (IA)  Strategy 

IA  is  defined  as  measures  that  protect  and  defend  information  and  information  systems 
by  ensuring  their  availability,  integrity,  authentication,  confidentiality,  and  nonrepudiation.  This 
includes  providing  for  restoration  of  information  systems  by  incorporating  protection,  detection, 
and  reaction  capabilities.  IA  must  be  addressed  for  all  weapon  systems;  command,  control, 
communications,  computers,  intelligence,  surveillance,  and  reconnaissance  systems;  and 
information  technology  programs  that  depend  on  external  infonnation  sources  or  provide 
information  to  other  DoD  systems. 

6.  Initial  Capabilities  Document  (ICD) 

The  ICD  is  required  before  MS-A,  as  part  of  the  JCIDS  process.  It  is  the  equivalent  of 
the  legacy  MNS,  but  is  much  more  definitive  in  describing  the  program  to  be  developed.  An 
ICD  documents  the  requirement  for  materiel  or  nonmateriel  approach,  or  an  approach  that  is  a 
combination  of  material  and  nonmateriel,  to  satisfy  specific  capability  gap(s).  It  defines  the 
capability  gap(s)  in  terms  of  the  functional  area,  the  relevant  range  of  military  operations,  desired 
effects,  time,  and  Doctrine,  Organization,  Training,  Material,  Leadership,  Personnel,  and 
facilities  (DOTMLFP),  education,  and  policy  implications  and  constraints.  The  outcome  of  an 
ICD  could  be  one  or  more  joint  DOTMLPF  change  Recommendations  (DCRs)  or  CDDs. 
Programs  that  enter  the  acquisition  process  at  MS-B  shall  have  an  ICD  that  provides  the  context 
in  which  the  capability  was  determined  and  approved,  and  a  CDD  that  describes  specific 
program  requirements.  Pre-MS-A  projects  shall  rely  on  the  ICD  as  the  basis  for  the  evaluation 
strategy. 
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7.  Capability  Development  Document  (CDD) 

The  CDD  is  required  at  MS-B  and  serves  the  same  purpose  as  the  legacy  ORD.  The 
CDD  outlines  an  affordable  increment  of  militarily  useful,  logistically  supportable,  and 
technically  mature  capability.  The  CDD  may  define  multiple  increments  if  there  is  a  sufficient 
definition  of  the  perfonnance  attributes  (key  perfonnance  parameters,  key  system  attributes,  and 
other  attributes)  to  allow  approval  of  multiple  increments.  The  CDD  is  required  at  Milestone  B 
for  new  programs  and  for  each  increment  of  an  evolutionary  acquisition  program. 

8.  Capability  Production  Document  (CPD) 

The  CPD  is  required  at  MS-C  as  part  of  the  JCIDS  process.  This  is  the  capabilities 
document  that  supports  IOT&E.  The  CPD  reflects  the  operational  requirements  resulting  from 
Engineering  Manufacturing  and  Development  and  details  the  perfonnance  expected  of  the 
production  system.  Software  shall  have  demonstrated  the  maturity  level  required  in  the  CPD 
prior  to  deploying  it  to  the  operational  environment.  Once  the  maturity  level  has  been 
demonstrated,  the  system  or  increment  is  base-lined,  and  a  methodical  and  synchronized 
deployment  plan  is  implemented  for  all  applicable  locations.  OT&E  shall  determine  the 
operational  effectiveness  and  suitability  of  a  system  under  realistic  operational  conditions, 
including  combat:  determine  if  thresholds  in  the  approved  CPD  and  COIs  have  been  satisfied; 
and  assess  impacts  to  combat  operations. 

9.  Reviewing  Foundation  Documents 

When  reviewing  these  documents  the  OTD  considers  from  an  operations  viewpoint,  why 
develop  it?  How  will  it  be  used?  In  what  installations  or  platfonns?  In  what  environments 
(natural  and  mandate)?  How  well  should  it  work?  When  should  it  work?  What  must  DT&E  and 
OT&E  do  to  prove  the  system’s  operational  effectiveness  and  suitability?  When  must  DT&E 
and  OT&E  prove  the  system’s  operational  effectiveness  and  suitability? 

10.  The  Test  and  Evaluation  Master  Plan  (TEMP) 

The  TEMP  is  the  single  most  important  T&E  document  associated  with  an  acquisition 
program.  It  is  the  controlling  T&E  management  document  for  all  acquisition  programs  and,  in 
general,  must  be  approved  by  the  Director  of  DT&E  and  the  Director  of  OT&E  at  milestone  B 
prior  to  beginning  OT&E.  The  TEMP  is  directive  in  nature,  and  defines  and  integrates  test 
objectives,  critical  issues,  system  characteristics,  test  responsibilities,  resource  requirements,  and 
test  schedules. 
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11.  Purpose  of  the  TEMP 

The  purpose  of  the  TEMP  is  to  combine  the  DA’s  DT&E  plans  and  COMOPTEVFOR’s 
OT&E  plans  into  one  integrated  master  plan  approved  by  the  CNO  or  higher  authority. 
COMOPTEVFOR  develops  the  Critical  Operational  Issues  (COIs)  for  each  program  and 
publishes  them  in  the  TEMP.  The  COIs  are  linked  to  CNO  requirements  established  in  the  ORD, 
ICD,  CDD,  or  CPD. 

Detennining  the  essential  elements  of  operational  effectiveness  and  operational 
suitability,  the  COIs  to  be  resolved  in  OT&E,  and  the  questions  which  must  be  answered  to 
resolve  those  issues  is  difficult;  a  contributing  factor  to  the  difficulty  is  the  number  of  sources  or 
agencies  involved.  MOEs  and  MOSs  should  be  clearly  established  in  the  requirements 
documents  COMOPTEVFOR  reviews  for  testability  and  appropriateness  prior  to  this  process. 
When  the  DA  provides  a  list  of  MOEs  and  MOSs  on  a  first-draft  TEMP,  the  OTD  ensures  that 
they  are  operational  characteristics,  not  technical  characteristics.  From  OPTEVFOR’s  viewpoint 
operational  backgrounds  overshadow  technical  backgrounds.  The  OTD  should  ask  two  basic 
questions  when  developing  input  to  the  TEMP: 

•  What  should  it  do  from  an  operational  mission  accomplishment  viewpoint? 

•  What  shouldn’t  it  do  from  an  operational  mission  accomplishment  viewpoint? 

Additionally  the  TEMP  serves  several  secondary  purposes.  It  allows  all  involved  to  see 

exactly  what  hurdles  the  system  must  clear  and  when,  it  allows  the  DA  to  project  T&E  costs 
which  must  be  funded,  and  it  allows  Fleet,  range,  simulator,  and  target  schedulers  to  plan,  well  in 
advance,  for  the  required  services.  Specifics,  including  requirements  of  new  or  modified 
facilities,  must  also  be  identified  in  the  TEMP. 

12.  TEMP  Policies 

The  contents  of  the  TEMP  and  the  relationship  of  key  portions  to  the  successful 
completion  of  the  overall  OT&E  program  cannot  be  overstated.  An  approved  TEMP  or  an 
approved  TEMP  revision,  constitutes  direction  to  conduct  the  specified  T&E  program,  including 
the  sponsor’s  committed  support,  and  constitutes  approval  of  the  COIs.  Test  plans  will  be 
prepared  directly  from  the  TEMP  and  will  carry  out  its  provisions. 

TEMPS  may  be  reviewed  in  their  entirety  twice:  once  when  the  DA  submits  a  draft  for 
comment,  and  again  when  the  final  version  is  received  for  the  Commander’s  signature.  Before 
the  first  review,  the  OTD  should  have  provided  the  DA  with  OT&E  schedule  inputs  for  Part  II,  a 
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complete  Part  IV,  and  OT&E  resource  requirements  fro  the  Part  V.  At  that  time,  OT-C  and  OT- 
D  should  be  included  in  the  schedule.  The  OTD’s  review  of  the  complete  TEMP  should  address 
all  parts.  The  DA  is  responsible  for  ensuring  the  TEMP  is  updated  at  milestones,  when  the 
program  baseline  has  been  breached,  or  on  other  occasions  when  the  program  has  changed 
significantly.  The  OTD  works  closely  with  the  DA  to  ensure  COMOPTEVFOR’s  input  is 
provided  in  sufficient  time  to  support  the  required  update,  ensuring  that  COMOPTEVFOR  is  not 
responsible  for  program  delays  while  preparing  TEMP  updates. 

The  TEMP  must  be  updated  at  all  milestones,  when  significant  program  changes  occur, 
or  when  the  program  baseline  has  been  breached. 

13.  TEMP  Basics 

A  TEMP  is  prepared  jointly  by  the  DA  and  COMOPTEVOR,  with  the  involvement  of 
both  the  OPNAV  program  sponsor  and  the  N091  T&E  coordinator  in  early  draft  reviews. 

During  the  TEMP  review  process,  the  OTD  ensures  the  minimum  acceptable  operational 
performance  requirements  (older  programs)  or  MOE/MOS  (newer  programs)  from  the  approved 
ORD/ICD/CDD/CPD  are  incorporated.  COMOPTEVFOR  contributes  to  all  parts  of  the  TEMP 
and  provides  the  OT&E  portions  throughout  the  document.  The  parts  specifically  provided  by 
COMOPTEVFOR  are  drafted  by  the  OTD. 

The  TEMP  is  required  at  MS-B  for  all  programs.  Since  the  TEMP  is  prepared  jointly  by 
the  DA  and  COMPOPTEVFOR,  the  OTD  is  involved  in  all  stages  of  TEMP  preparation.  This 
requires  the  OTD  to  be  familiar  with  other  program  documentation  (MNS,  ORD, 

ICD/CDD/CPD,  Infonnation  Assurance  (IA)  strategy,  ONI  Capstone  TA,  etc.)  and  close 
coordination  with  the  DA,  particularly  during  program  changes. 

14.  Test  Planning 

Operational  testing  consists  of  exercising  a  system  or  equipment  under  conditions  that  are 
as  close  as  possible  to  the  expected  natural,  operational,  and  combat  environment  using 
operational  scenarios.  Forces,  friendly  and  oppositional,  apply  realistic  tactics  against  targets 
that  fight  back.  Additionally,  operational  testing  consists  of  ensuring  that  the  test  article  is 
representative  of  the  intended  production  equipment  and  is  installed  as  it  is  expected  to  be 
installed  in  the  Fleet.  The  test  article  is  operated  and  usually  maintained  by  Fleet  personnel. 

OT  provides  data  on  system  perfonnance  in  the  operational  environment.  Performance 
includes  all  the  elements  of  operational  effectiveness  and  operational  suitability.  The 
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environment  includes  many  things  such  as  people  (operators,  maintainers,  etc.);  other  systems 
that  will  also  be  consuming  power,  radiating,  etc.,  in  the  same  ship  or  aircraft;  ships  or  aircraft  in 
the  vicinity,  employing  their  own  systems;  established  constraints  or  rules  of  engagement; 
natural  environmental  factors  (visibility,  sea  state,  oceanographic,  etc.);  the  simulated  enemy, 
tactics,  and  countermeasures  the  operator  employs;  and  so  on. 

15.  Test  Plan  Preparation 

Test  plans  are  required  for  each  identified  phase  of  OT&E  for  all  ACAT  programs. 
Preparing  the  test  plan  focuses  on  several  fundamental  issues  important  to  the  overall  OT&E 
process.  These  issues  include  the  purpose  of  the  test;  capabilities/functions  of  the  COIs  to  be 
examined;  how  the  test  will  be  conducted,  whether  operation-oriented  or  scenario-oriented 
testing  will  be  used;  evaluation  criteria  against  which  test  results  will  be  measured;  resources 
required  to  support  OT&E;  data  collection  methods  and  requirements;  and  data  analysis  methods 
to  be  employed.  Test  plan  writing  begins  upon  resolution  of  these  issues. 

16.  MBTD  Planning  Process 

The  OT&E  methodology  is  moving  from  functional-based  test  design  to  Mission-Based 
Test  Design.  Regardless  of  the  stage  in  the  acquisition  process  a  program  resides,  the  initial 
steps  in  the  MBTD  process  are  alike  once  the  decision  has  been  made  to  use  the  MBTD 
methodology.  The  MTBD  process  is  also  applicable  to  FOT&E.  The  MBTD  planning  process 
is  followed  in  its  entirety  for  each  increment  or  spiral  of  a  program,  although  some  portions  of 
the  process  may  be  abbreviated. 

Figure  32  from  COMOPTEVFOR  Instruction  3980.1  [2008]  reflects  an  overview  of  the 
MBTD  and  OT  Framework  development  process.  This  figure  depicts  the  steps  of  a  mission 
analysis  for  the  system  under  test.  It  shows  the  steps  required  to  develop  the  infonnation  needed 
to  write  the  OT  Framework.  It  also  depicts  how  the  OT  Framework  acts  as  input  to  the  IT 
integration  process  (if  applicable)  and  how  the  mission  analysis  provides  the  OTD/C  the 
capability  to  trace  the  results  of  testing  back  to  the  mission  offering  bidirectional  traceability. 
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MBTD/IT  Construct 


Figure  32.  MBTD  (and  IT  when  applicable)  Process  Flow  Chart  “From 
COMOPTEVFOR  Instruction  3980. 1  [2008].” 


The  MBTD  process  begins  with  a  meeting  or  series  of  meetings  held  between  the 
program  T&E  stakeholders.  The  intent  of  these  meetings  is  to  [COMOPTEVFORINST  3980.1: 
6-3]: 

•  Define  and  map  the  overarching  T&E  strategy  to  ensure  all  stakeholders  start  from  the 
same  philosophical  foundation. 

•  Develop  a  list  of  acquisition  and  operational  documents  required  to  support  the  mission 
analysis  effort. 

•  Determine  who  should  participate  in/support  the  mission  analysis  effort  and  where  and 
when  it  will  commence. 

•  Define  the  format  and  content  of  testing  requirements  inputs  to  an  IT  matrix  database  (if 
IT  is  applicable). 

•  Determine  where  the  IT  matrix  database  (if  IT  applicable)  will  be  maintained  and  who 
will  have  access. 
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•  Derive  test  data  and  information  sharing  rules  (may  be  adjusted  for  the  ITT  charter). 

•  Establish  separate  analysis/reporting  requirements. 

The  ground  rules  and  definitions  from  the  meetings  are  documented  in  the  approved 
TEMP.  The  ground  rules  may  be  incorporated  in  a  Memorandum  of  Agreement  (MO A) 
approved  at  an  appropriate  level  in  the  event  that  the  TEMP  or  a  TEMP  update  is  too  far  in  the 
future.  The  ground  rules  and  definitions  are  later  followed  by  approval  in  the  TEMP. 

17.  Mission  Analysis 

The  purpose  of  mission  analysis  is  to  identify  the  tasks  the  system  will  support,  derive  the 
mission-based  COIs,  partition  the  tasks  into  subtasks,  and  correlate  the  result  with  the  required 
capabilities  and  their  associated  attributes  from  the  ORD/Capabilities  Document  (CD).  Upon 
completion  of  these  tasks  direct  traceability  of  any  system  enhancement,  risk  area,  or  deficiency 
discovered  during  testing  to  a  mission  or  missions  is  enabled. 

Mission  analysis  is  a  combined  effort  among  COMOPTEVFOR,  the  program 
representatives,  and  other  participants  such  as  the  Fleet  Forces  Command  (N8)  representatives 
and  operational  users.  Additional  Subject  Matter  Experts  (SME)  may  be  included  to  ensure  this 
evolution  is  completed  thoroughly.  Inclusion  of  key  participants  ensures  DT,  OT,  and  CT 
stakeholders  (when  included)  are  in  full  agreement  concerning  the  missions,  tasks,  requirements, 
and  defined  capability  attributes  of  the  system.  To  ensure  all  T&E  members  are  working  from 
the  same  foundation,  key  participants  must  first  complete  and  agree  on  the  mission  analysis  for 
the  system.  The  mission  analysis  is  conducted  by  the  participants  to  derive  the  tasks  and 
missions  applicable  to  the  system  to  be  tested  using  the  following  system  documents: 

•  The  CONOPS  or  concept  of  employment 

•  The  current  ORD/CD 

.  The  Universal  Navy  Task  List  (UNTL)  (OPNAVINST  3500.38B) 

•  The  Universal  Joint  Task  List  (UJTL),  (CJCSM  3500. 04C)  (for  joint  programs) 

•  The  Navy  or  Marine  Mission  Essential  Task  List  (METL) 

•  Other  appropriate  documents 

18.  Detailed  Process  Description 

Mission  analysis  consists  of  6  steps:  identify  tasks;  derive  mission  COIs;  identify 
subtasks;  establish  conditions;  develop  attribute  matrix;  and  allocate  COIs,  tasks,  and  subtasks  to 
attributes.  Mission  analysis  is  an  iterative  process  where  two  or  three  of  these  steps  will  be 
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conducted  together  and  repeated  several  times  to  complete  the  breakdown  of  a  given  mission  into 
tasks,  subtasks,  and  conditions.  When  the  mission  analysis  is  complete,  the  results  must  be 
documented  in  the  OT  Framework  and  approved  in  the  TEMP  (or  MOA  if  needed). 

Step  One:  Identify  Tasks.  This  step  begins  by  conducting  an  analysis  of  available 
documents  (ORD/CD,  CONOPS,  UNTL,  and  METL  for  units  employing  the  system  under  test) 
to  identify  tasks  that  the  system  under  test  will  support.  There  are  2  general  cases  to  this  task. 
The  first  case  is  when  the  Mission  Essential  Tasks  (MET)  that  the  system  will  support  are  known 
and  well  documented.  The  second  is  when  the  specific  MEATs  are  not  adequately  documented. 

Step  One,  Case  1:  The  METs  that  the  system  will  support  are  known  and  well 
documented.  This  is  usually  documented  in  the  CONOPS  section  of  the  CD  or  some  other 
task  lists.  The  known  METs  are  collected  and  sorted  into  categories  as  either  “mission  tasks” 
or  “supporting  tasks.”  The  definition  of  mission  tasks  is  highly  dependent  upon  the  system 
and  the  unit  employing  it.  Supporting  tasks  include  those  tasks  that  support  a  defined  mission 
task  or  multiple  mission  tasks. 

Some  tasks  may  not  be  called  out  in  available  documentation.  Upon  completion  of 
sorting  the  METs  noted  above,  the  UNTL  is  reviewed  to  determine  if  there  are  other  tasks 
that  would  fit  the  system  under  test.  If  additional  tasks  are  identified,  agreement  between  all 
stakeholders  is  sought  early  in  the  process  to  ensure  that  all  selected  tasks  are  appropriate  for 
test. 

Step  One,  Case  2:  In  the  case  when  specific  METs  are  not  adequately  documented 
there  may  be  no  direct  relationships  established  to  the  UNTL.  Consequently,  all  available 
requirements  documents  must  be  analyzed  to  determine  what  missions  the  system  might 
support.  Once  a  notional  mission  list  is  compiled,  the  UNTL  is  then  reviewed  and  appropriate 
UNTL  tasks  are  “mapped”  to  the  notional  mission  list. 

Additionally,  reviewing  the  UNTL  may  yield  unique  mission  tasks  not  found  in  other 
documents  and/or  supporting  tasks  that  could  be  considered  for  multiple  missions.  Once 
tasks  have  been  selected,  agreement  between  all  stakeholders  is  sought  early  in  the  process  to 
ensure  that  all  selected  tasks  are  appropriate  for  test. 

Upon  completion  of  step  one  the  OTD  should  have  a  documented  list  of  all  mission  and 
supporting  tasks. 
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Step  Two:  Derive  Mission  COIs.  The  mission  COIs  are  derived  next  based  on  the  list  of 
mission  tasks  identified  in  step  one.  COIs  are  created  from  individual  mission  tasks  or  from 
grouping  similar  mission  tasks  to  fonn  a  single  COI.  It  is  important  to  ensure  that  tasks  are 
combined  such  that  the  COI  is  specific  enough  to  be  relevant  to  the  unit  using  the  system,  but  not 
so  specific  that  each  COI  is  simply  a  variation  of  an  overarching  mission.  Upon  completing 
additional  steps  in  the  MBTD  process,  the  similarities  in  task  breakdowns,  conditional  variations, 
and  capabilities  correlation  that  exist  between  COIs  may  become  more  obvious  and  the  COI  list 
may  need  to  be  adjusted.  The  result  of  this  step  is  a  list  of  specific  COIs  written  in  a  particular 
format.  Additionally,  a  COI/task  summary  and  a  COI  task  breakdown  hierarchy  is  developed. 

Step  Three:  Identify  Subtasks.  Subtasks  are  separate  actions  to  be  performed  to  carry 
out  each  task.  It  is  important  to  breakdown  the  subtasks  to  the  correct  level  as  the  subtask 
breakdown  will  enable  the  identification  of  all  conditions  (in  step  four)  that  may  potentially 
impact  task  perfonnance.  It  will  also  allow  for  correlation  of  capability  attributes  to  specific 
sub  tasks. 

Temporal  views,  block  diagrams  that  depict  the  steps  of  the  task  process  in  order  of 
occurrence,  are  produced  for  each  task  to  depict  the  breakdown  of  the  task  into  its  component 
subtasks.  The  numbering  system  established  in  step  two  is  further  expanded  to  include  the 
subtasks  in  order  to  create  a  hierarchical  numbering  of  each  mission-task-subtask  breakdown.  At 
the  completion  of  step  three,  the  OTD  should  have  a  temporal  view  for  each  task,  a  mission-to- 
subtask  hierarchical  list,  and  a  subtask  cross-reference  table. 

Step  Four:  Establish  Conditions.  Conditions  are  variables  of  the  operating 
environment  that  affect  the  perfonnance  of  the  subtask;  they  describe  the  physical  (littoral,  open 
ocean,  calm  seas,  low  visibility,  etc.),  military  (single  unit/task  force/joint  operations,  aircraft 
division,  etc.),  and  civil  (population  density,  civil  unrest,  etc.)  variations  that  impact  subtask 
perfonnance  and  fonn  the  operational  context  for  selected  subtasks.  Conditions/descriptors 
should  be  derived  or  implied  for  the  ORD/CD.  If  the  more  general  UNTL  conditions/descriptors 
need  to  be  used,  they  should  be  modified  to  fit  the  system-specified  capabilities. 

Once  the  conditions  are  established  and  documented,  the  informational  view,  a  temporal 
view  block  diagram  with  conditions,  inputs,  and  outputs  added  to  it,  is  produced  using  the 
temporal  view  (from  step  three).  First  applicable  conditions  for  each  subtask  are  added.  Once 
conditions  have  been  added,  subtask  inputs  and  outputs  are  added.  At  the  completion  of  this  step, 
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the  OTD  should  have  produced  a  conditions  directory  listing  applicable  conditions  for  all 
subtasks  and  an  informational  view  for  each  task. 

Step  Five:  Develop  Attribute  Matrix.  Document  capability  attributes  from  the 
ORD/CD  in  matrix  fonnat.  An  attribute,  sometimes  referred  to  in  legacy  documents  as  a 
“requirement,”  is  as  a  quantitative  or  qualitative  characteristic  of  an  element  or  its  actions,  where 
element  refers  to  the  system.  Qualitative  attributes  may  be  expressed  in  terms  of  an  action  or 
outcome  required  from  the  system.  Quantitative  attributes  usually  have  numbers  associated  with 
them  and  may  have  quantifiable  standards. 

Once  all  attributes  have  been  documented,  standards  are  defined.  Standards  may  be 
provided  in  the  ORD/CD.  In  cases  a  standard  has  not  been  defined  in  the  ORD/CD,  one  must  be 
defined.  If  possibly  a  quantitative  standard  is  applied  to  a  qualitative  attribute.  If  this  is  not 
possible  the  attribute  matrix  must  somehow  specify  what  is  “good  enough”  with  regard  to  the 
attribute,  even  if  a  survey  is  used  as  the  data  collection  method. 

At  the  completion  of  step  five,  the  OTD  should  have  a  table  or  spreadsheet  of  all 
ORD/CD-defined  attributes  and  their  associated  standards.  This  matrix  will  fonn  the  basis  for 
subsequent  steps  as  additional  information  is  added,  and  will  end  up  forming  the  basis  for  the 
OT-DT-CT  input  to  the  integration  effort  that  occurs  after  framework  development. 

Step  Six:  Allocate  COIs,  Tasks,  and  Subtasks  to  Attributes.  This  step  continues  the 
population  of  the  attribute  matrix  created  in  step  five.  It  takes  the  documented  ORD/CD 
attributes  and  their  associated  standards,  and  linking  COIs,  tasks,  and  subtasks  to  them.  Many 
linkages  are  explicitly  defined  in  the  ORD/CD.  For  those  that  are  not,  linkages  will  have  to  be 
determined  based  on  operational  experience.  Many  attributes  may  be  linked  to  multiple  missions, 
tasks,  and  subtasks,  further  complicating  the  process.  However,  the  linkages  must  be  made  as 
they  are  absolutely  essential  to  the  direct  traceability  between  attributes/standards  (including  Key 
Performance  Parameters  (KPP)  and  thresholds)  and  mission  COIs. 

At  the  completion  of  this  step,  the  attribute  matrix  developed  in  step  five,  and  further 
populated  with  COI  and  subtask  information,  should  form  the  basis  of  the  attribute-to-subtask 
matrix.  If,  at  the  end  of  step  six,  it  is  detennined  that  an  attribute  from  the  list  cannot  reasonably 
be  associated  to  a  COI,  task,  or  subtask,  consideration  may  be  given  to  removing  it  from  the  list 
as  either  an  orphan  attribute  or  as  a  system  attribute  that  has  little  or  no  relation  to  the  operational 
effectiveness  or  suitability  of  the  system.  The  program  sponsor  and  other  members  of  the  ITT 
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should  be  consulted  prior  to  making  a  final  decision  with  regard  to  removal  of  an  attribute  from 
the  list  since  this  database  will  be  shared  by  T&E  stakeholders  within  the  ITT. 

Once  the  above  steps  are  complete  the  mission  analysis  section  is  complete.  This 
provides  the  prerequisites  for  stand-alone  OT  planning.  If  consensus  between  all  T&E 
stakeholders  on  the  mission  analysis  for  the  system  is  not  achieved  the  statutory  requirement  for 
OT  independence  dictates  that  further  OT  planning  be  based  on  the  OT  position  regarding  the 
specific  disagreement.  The  product  of  this  mission  analysis  effort,  including  the  templates  and 
the  database,  must  be  documented  (to  confirm  agreement  among  the  participants)  in  the 
evaluation  strategy  or  TEMP.  If  the  TEMP  is  still  months  away,  an  MO  A  can  be  used  in  the 
interim. 

19.  Completing  the  OT  Framework/Stand-Alone  OT  Planning 

Additional  steps  are  perfonned  by  the  OTD  to  complete  the  OT  Framework.  Note  that 
once  the  mission  analysis  is  complete  and  documented,  OT  and  DT  (and  CT  at  some  point)  can 
begin  separate  test  planning.  Separate  planning  is  conducted  to  produce  the  stand-alone  test 
objectives  for  OT  and  DT  (and  CT,  when  appropriate). 

OT  and  DT  planners  document  their  test  objectives,  prior  to  integrating,  to  ensure: 

•  The  IT  and  independent  OT  will  provide  OPTEVFOR  sufficient  data  and  assurance  in  the 
results  of  testing  to  make  an  effectiveness  and  suitability  detennination,  and  Fleet 
release/Fleet  introduction  recommendation. 

•  A  basis  is  established  for  calculating  savings/cost  avoidance  attributable  to  the  IT  effort. 

•  Each  entity  (OT,  DT,  and  CT)  has  an  adequate  and  approved  framework  for  their  testing 
and  the  integration  process;  for  oversight  programs,  this  would  include  DOT&E  approval 
of  the  OT  Framework. 

The  separate  OT  planning  begins  with  the  products  of  the  mission  analysis  effort. 

Step  Seven:  Develop  Additional  Operational  Attributes  and  Standards.  The  OTD 

expands  on  previous  steps  by  developing  operational  attributes  which  were  not  included  in  the 
ORD/CD;  developing  associated  standards  for  these  attributes;  and  tying  them  to  COIs,  tasks, 
and  subtasks.  By  developing  a  list  of  operational  attributes  to  be  used  in  conjunction  with  the 
ORD/CD  documented  attributes,  the  OTD  helps  to  more  clearly  focus  the  testing  toward 
answering  the  COI  questions  so  as  not  to  become  a  mere  “spec  checker.”  At  the  completion  of 
the  process,  the  OTD  should  have  a  comprehensive  attribute-to-subtask  matrix  that  includes  all 
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necessary  test  attributes.  This  comprehensive  matrix  is  not  intended  to  ensure  all  requirements 
and  operational  expectations  for  the  system  are  defined  prior  to  the  beginning  of  IT. 

Step  Eight:  Devise  Test  Methods  for  Each  Subtask.  In  steps  eight  and  nine  the  OTD 
devises  test  methods  for  each  sub  task,  keeping  in  mind  the  attributes/standards  and  data 
collection  requirements  associated  with  these  subtasks.  Although  these  tasks  are  listed  as 
separate  steps,  they  are  completed  simultaneously.  Test  methods  describe  the  approach  to  be 
used  to  collect  the  data  necessary  to  determine  the  satisfactory  execution  of  the  subtask,  and  to 
detennine  performance  as  compared  to  the  derived  and  implied  standards.  Consequently,  data 
collection  requirements  must  be  known  before  test  methods  can  be  finalized.  When  completed, 
the  test  methods  will  form  one  of  the  building  blocks  for  the  vignette  descriptions  built  in  step 
ten. 

The  OTD  uses  the  subtask  cross-reference  matrix  developed  in  step  three  to  begin 
devising  test  methods  for  each  subtask.  This  matrix  ties  all  like  subtasks  together  to  enable  the 
OTD  to  develop  a  common  method  for  common  subtasks.  The  OTD  adds  “Attribute,”  “Test 
Method,”  and  “Data  Requirements”  columns  to  the  matrix.  Using  the  matrix  from  step  seven,  the 
attribute  numbers  corresponding  to  the  subtasks  should  be  added.  The  test  method  is  added  to  the 
matrix  for  each  subtask  once  it  has  been  developed.  In  conjunction  with  step  nine,  data 
requirements  are  added  to  the  final  column.  The  goal  of  this  step  is  to  ensure  the  collection  of 
the  data  necessary  to  detennine  the  satisfactory  execution  of  the  subtask  and  to  determine 
performance  as  compared  to  the  attributes/standards. 

Step  Nine:  Derive  Data  Requirements.  Derive  data  requirements  for  all  subtasks  to 
detennine  whether  the  COI,  task,  and  subtask  have  been  satisfactorily  accomplished,  and 
whether  all  standards,  quantitative  and  qualitative,  have  been  met.  Describe  quantitative  and 
qualitative  data  requirements  in  sufficient  detail  to  support  the  integration  process.  For 
qualitative  data,  the  OTD  first  defines  what  qualitative  infonnation  is  needed  from  the  survey  to 
support  the  resolution  of  COIs  and  standards  before  relevant,  useful  surveys  can  be  constructed. 
At  the  completion  of  steps  eight  and  nine,  the  OTD  should  have  a  test  method/data  requirements 
matrix  that  directly  ties  data  elements  back  through  test  methods,  standard,  subtasks,  tasks,  and 
COIs. 

Step  Ten:  Build  Vignettes.  The  OTD’s  team  (analyst,  other  operational  testers)  takes  all 
of  the  infonnation  developed  in  steps  one  through  nine  and  builds  test  “vignettes.”  Vignettes  are 
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designed  to  ensure  the  thorough  testing  of  all  attributes,  standards,  tasks/sub  tasks,  and  missions. 
Creating  vignettes  consists  of  parsing  mission  and  task  execution  into  manageable  chunks  which 
can  be  accomplished  within  a  dedicated  OT  period  or  throughout  the  longer  IT  period.  Vignette 
descriptions  provide  the  test  team  with  a  detailed  methodology  and  “game  plan”  for  the 
execution  of  individual  test  events  to  ensure  all  OT  objectives  are  met. 

When  involved  in  the  IT  process,  the  OTD  uses  the  vignette  development  process  to 
detennine  how  much  independent  OT  is  required  at  the  completion  of  IT.  Executing  the  IT 
vignettes  provides  much  of  the  information  necessary  to  resolve  COIs  prior  to  the 
commencement  of  IOT&E  at  the  end  of  the  IT  phase.  IOT&E  becomes  a  series  of  end-to-end 
mission  confirmation  events  based  around  combined  IT  vignettes. 

Vignettes  are  constructed  from  single  sub  tasks  in  some  cases  and  in  others  sub  tasks  can 
be  grouped  together  logically  to  be  executed  together.  The  OTD  attempts  to  minimize  the 
number  of  overall  conditions,  developed  in  step  four,  associated  with  the  task  and  its  component 
subtasks.  In  this  step,  the  condition  list  should  be  reviewed  and  narrowed  down  to  include  only 
those  conditions  which  impact  the  perfonnance  of  the  system  under  test  in  relation  to  the 
subtask. 

The  OTD  decides  on  appropriate  subtask  parsing  and  develops  basic  vignette 
descriptions.  The  vignettes  descriptions,  their  associated  identifiers,  subtasks,  and  conditions  are 
then  combined  into  a  vignette-to-conditions  matrix.  Typically  there  will  be  many  sub  tasks  and 
associated  conditions,  multiple  matrices  may  be  needed.  The  end  result  this  process  should 
provides  the  OTD  with  a  basic  description  of  the  vignettes,  which  will  be  used  to  create  more 
detailed  vignette  procedures. 

The  OTD  identifies  each  vignette  as  a  candidate  for  integration  or  as  an  independent 
OAT  or  IOT&E  when  planning  for  IT.  This  facilitates  the  integration  process  and  provides 
fidelity  to  the  OT  Framework. 

The  test  method  and  data  requirements,  derived  in  steps  eight  and  nine,  supports  the 
development  of  detailed  vignette  procedures  by  adding  fidelity  to  the  vignette  description.  The 
output  of  step  ten  is  a  vignette-to-conditions  matrix,  a  vignette  cross-reference  matrix,  and  a 
vignette  procedures  matrix  to  which  resource  requirements  will  be  added  in  step  eleven. 

Step  Eleven:  Determine  Resource  Requirements.  Upon  completing  the  previous  steps 
the  OT  team  has  all  the  infonnation  required  to  determine  test  resource  requirements  for  stand- 
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alone  OT  of  the  system.  In  this  step  the  OTD  documents  these  requirements  in  sufficient  detail  to 
support  either  dedicated  OT  or  the  IT  integration  process,  including: 

•  Number  of  test  articles  with  any  specific  configuration  requirements 

•  Specific  aircraft,  ship,  submarine,  unit,  or  exercise  support  requirements 

•  Specific  range,  test  site,  instrumentation,  and  threat  requirements 

•  Flight  hours,  at-sea  time,  or  system  operating  time 

•  Special  support  equipment  requirements 

•  Any  M&S  requirements 

•  Specific  operator  or  maintenance  training  requirements 

•  Pre-faulted  modules  or  Maintenance  Demonstrations  (M-DEMO) 

•  Any  special  instrumentation  or  data  collection  requirements 

These  resources  are  identified  by  vignette  and  then  rolled  up  to  detennine  the  overall 
stand-alone  OT&E  requirements.  The  OTD  also  identifies  any  potential  limitations  to  test  for 
inclusion  in  the  OT  Framework  in  this  step.  Examples  include  threat  replication,  inability  to  test 
the  system  in  certain  environments  that  were  identified  as  significant  conditions  in  step  ten,  or 
unavailability  of  key  test  resources  or  instrumentation. 

Once  the  step  is  completed  the  OTD  can  now  bi-directionally  trace  a  data  element  all  the 
way  up  to  the  mission  COI  based  on  the  requirements/capabilities-to-subtask  correlation  already 
completed  in  the  mission  analysis  effort.  The  OTD  can  now  write  an  OT  Framework,  with  the 
key  component  being  the  final  vignette  matrix  developed  through  this  process. 

20.  OT  Framework 

The  OT  Framework  is  the  primary  document  that  defines  adequate  OT  and,  when 
appropriate,  for  integrating  the  OT  objectives  with  DT  and  CT  objectives  to  form  an  IT  matrix. 
The  OT  Framework  also  provides  the  basis  for  the  OT  input  to  the  TEMP,  and  defines  the  OT 
objectives  and  the  specific  test  requirements  for  resolution  of  each  COI,  and  the  OTD’s 
minimum  IOT&E  test  requirements.  The  OT  Framework  is  reviewed  and  changed  if  necessary 
any  time  there  are  significant  program  or  documentation  changes/revisions  since  it  is  generated 
much  earlier  in  the  T&E  process  timeline.  Changes  include  completion  of  CDR,  the  release  of  a 
CDD  or  CPD,  or  any  major  program  perturbations.  An  update  or  change  to  the  OT  Framework 
is  an  appropriate  place  to  document  any  limitations  to  test  that  may  arise  during  the  course  of  the 
IT  effort. 
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The  OT  Framework  must  be  approved  by  COMOPTEVFOR  and  DOT&E  for  oversight 
programs.  Once  approved,  the  OTD  is  ready  to  begin  the  integration  process.  Any  future 
changes  to  the  OT  Framework  would  be  handled  in  the  same  way  changes  to  an  approved  test 
plan  are  handled. 

21.  Test  Integration 

IT  combines  CT,  DT,  and  OT  to  form  a  cohesive  testing  continuum.  Participants  (CT, 

DT,  and  OT)  must  have  detennined  their  entering  objectives  for  adequate  testing  of  the  system 
under  evaluation  before  integration  testing  can  occur.  IT  does  not  alleviate  the  requirement  for 
independent  OT  reporting  based  on  separate  OPTEVFOR  analysis  of  the  shared  test  information 
produced  by  the  IT  effort. 

The  integration  process  begins  with  the  OT  Framework  approved.  The  goal  of  the  process 
is  to  identify  any  and  all  opportunities  for  synergy  in  planning,  execution,  and  data  collection 
during  the  IT  period.  However,  from  an  OT  perspective,  an  identified  synergy  may  be  lost  if  the 
system  configuration  changes  at  a  later  date  or  the  data  collected  is  deemed  unusable  for  some 
other  reason.  Each  entity  should  enter  the  process  with  a  matrix  of  test  objectives  in  a  compatible 
format  and  based  on  an  agreed  mission  analysis  structure.  The  first  accomplishment  is  preparing 
and  obtaining  approval  of  an  Integrated  Test  Team  (ITT)  Charter.  The  charter  specifies  critical 
coordination  factors  such  as  [COMOPTEVFORINST  3980.1:  6-19]: 

•  IT  matrix  development  and  format  for  OT,  DT,  and  CT  inputs 

•  Detailed  IT  event  planning  and  execution  process 

•  Data/test  information  sharing  criteria 

•  Separate  analysis/reporting 

•  Data  format  and  handling 

•  Data  repository  location 

•  Data  fidelity  requirements 

•  Scoring  criteria  and  formula  for  calculated  metrics 

•  Process  for  arbitration  of  disputes 

•  Process  for  inclusion  of  supplemental  or  regression  testing  requirements 

•  Process  for  prioritization  of  testing  requirements 

•  Method  for  identification  of  comparative  cost  savings/schedule  compression  as  a  result  of 

IT 
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The  ITT  should  stand  up  soon  after  contract  award,  which  ensures  OT  participation  early 
in  the  development  of  the  system  under  test.  The  product  of  the  IT  integration  effort  is  an  IT 
database,  similar  in  structure  and  content  to  the  OT  Framework  database,  but  merged  with  DT 
and  CT  objectives. 

22.  OPTEVFOR  Test  Plans 

COMOPTEVFOR  test  plans  consist  of  E-tests  and  S-tests.  E-tests  are  keyed  to  the  COIs 
and  are  given  the  title  of  the  COIs  they  are  intended  to  address.  There  are  10  S-tests  which  are 
standardized  in  COMOPTEVFOR  test  plans.  They  are: 

(1)  Test  S-l,  Reliability 

(2)  Test  S-2,  Maintainability 

(3)  Test  S-3,  Availability 

(4)  Test  S-4,  Logistic  Supportability 

(5)  Test  S-5,  Compatibility 

(6)  Test  S-6,  Interoperability 

(7)  Test  S-7,  Training 

(8)  Test  S-8,  Human  Factors 

(9)  Test  S-9,  Safety 

(10)  Test  S-10,  Documentation 

All  of  the  10  standard  S-tests  will  usually  be  applicable  to  IOT&E.  Some  may  not  be 
appropriate  to  very  early  OT&E  phases  or  to  late  FOT&E.  The  inappropriate  tests  are  omitted  in 
these  cases.  Additional  tests  are  used  as  required. 

23.  COIs  and  Evaluation  Criteria 

Each  phase  of  OT&E  is  an  investigation  of  operational  effectiveness  and  operational 
suitability  of  the  system  up  to  that  point  in  time.  Prior  to  IOT&E,  in  the  early  phases,  COIs  are 
assessed  by  evaluating  risks  associated  with  each  COI.  In  IOT&E,  COIs  are  resolved  as  either 
SAT  or  UNSAT,  or  are  unresolved. 

The  essential  elements  of  operational  effectiveness,  the  things  the  system  must  and  must 
not  do  for  mission  accomplishment,  vary  from  one  system  to  the  next.  For  a  given  system,  the 
essential  measures  of  operational  effectiveness  and  operational  suitability  form  the  framework 
for  the  capabilities  and  functions  of  the  COIs;  the  COIs  define  operational  effectiveness  and 
operational  suitability  for  a  given  program. 
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All  COI  capabilities  and  functions  are  examined  in  IOT&E.  Not  all  COI  capabilities  and 
functions  are  examined  during  earlier  OT&E.  This  is  because,  at  this  point,  the  equipment 
generally  does  not  closely  approximate  the  planned  production  configuration.  The  capabilities 
and  functions  of  COIs  for  a  phase  of  OT&E  are  documented  in  the  TEMP. 

24.  Scenario-Oriented  or  Operation-Oriented  Testing 

Although  the  OT&E  testing  methodology  is  moving  to  MBTD,  it  is  not  always  the  way  to 
perfonn  OT  testing.  Two  common  methods  in  OT&E  when  not  doing  MBTD  are  scenario- 
oriented  testing  and  operation-oriented  testing. 

Scenario-oriented  testing  is  typically  employed  for  systems  whose  modes  of  operation  or 
functions  change  in  response  to  a  changing  operational  situation  (e.g.,  a  radar  suite).  The  OTD 
develops  scenarios  based  upon  the  threat  as  derived  from  the  applicable  threat  documents  to 
stress  the  system  under  test  in  a  realistic  threat-representative  manner.  Scenario-oriented  testing 
typically  allows  the  Fleet  user  the  greatest  flexibility  in  operating  the  system  as  the  tactical 
situation  changes.  This  affords  the  OTD  greater  opportunity  to  make  informed  decisions  on  the 
merits  of  the  system  and  its  capability  to  meet  CNO-assigned  thresholds. 

When  developing  the  scenarios,  the  OTD  must: 

•  Be  complete  and  state  what  results  are  expected  from  each  scenario. 

•  Describe  the  tactical  situation  at  the  start  of  the  exercise  (e.g.,  single-ship  littoral 
operations  with  a  high  probability  of  air  attack). 

•  Describe  the  situation  that  develops  (e.g.,  electronic  surveillance  detection  of  enemy 
aircraft)  and  what  is  expected  to  happen  (e.g.,  detection,  acquisition,  and  engagement  of 
the  enemy  aircraft). 

•  Supplement  this  narrative  with  diagrams  or  tables  specifying  the  movements  of  exercise 
participants  (friendly  and  enemy)  and  their  expected  actions  at  specific  times. 

•  Develop  a  sufficient  number  of  scenarios  to  test  the  system,  and  be  prepared  for  the 
unexpected.  Commanding  officer’s  tactical  decisions,  loss  of  assets  or  services,  or  fouling 
of  the  firing  ranges  can  all  lead  to  unexpected  results  or  non-completion  of  scenarios. 
Several  scenarios  may  be  required  for  multipurpose  systems  to  exercise  their  various 

capabilities.  The  data  recorded  during  the  scenarios  are  used  for  reconstruction  and  analysis  of 
the  various  E-tests  and  S-tests.  Often,  scenario-oriented  testing  is  dedicated  testing  (in  tenns  of 
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Fleet  RDT&E  support),  although  it  can  be  accomplished  on  a  not-to-interfere  basis  during  Fleet 
exercises. 

Operation-oriented  testing  is  typically  employed  for  equipment  whose  mode  of  operation 
or  function  does  not  change  with  the  tactical  situation  (e.g.,  torpedo  tubes  or  waste  disposal 
systems).  These  systems  are  either  in  use  or  not  in  use.  They  can  be  tested  by  simply  operating 
them  in  the  anticipated  environment.  The  events  and  conditions  necessary  during  system 
operation  must  be  specified  (e.g.,  the  targets  and  operating  environments)  when  operation- 
oriented  testing  is  used.  Test  events  and  conditions  must  provide  an  operationally  realistic  test  of 
the  system. 

In  either  scenario-  or  operation-oriented  testing,  the  following  are  kept  in  mind: 

Simulation  of  all  possible  enemy  actions  are  included  is  included  in  testing.  This 
includes  countenneasures  to  tactics  employed.  All  reasonable  expected  actions  the  target 
systems  are  expected  to  encounter  must  be  adequately  replicated  in  the  test.  The 
replication  is  representative  of  enemy  capabilities.  The  range  of  environments  and 
threats  is  covered. 

The  natural  and  electronic  test  environment  should  approximate  the  anticipated 
operating  environment.  This  includes  the  anticipated  background  noise  caused  by  other 
ships,  aircraft,  etc.,  to  determine  the  effects  of  Electromagnetic  Interference  (EMI);  the 
anticipated  natural  environmental  conditions,  such  as  sea  state,  temperature,  cloud  cover, 
etc.,  to  enable  a  detennination  of  their  effect  on  system  perfonnance;  operation  of  other 
equipment  that  may  be  used  in  conjunction  with  the  tested  equipment  to  allow  evaluation 
of  changes  in  electrical  power  loads,  effects  of  gunfire-induced  shock  and  vibration,  EMI, 
etc.;  and  all  relevant  joint  interfaces  and  other  interfaced  units/systems  in  the  anticipated 
joint  operating  environment. 

The  number  of  resources  required  for  testing  should  reflect  what  the  weapon 
system  would  realistically  be  expected  to  encounter  in  actual  operations. 

25.  Operational  Test  Director  Responsibilities 

The  Operational  Test  Director  has  responsibilities  prior  to  and  during  test  operations. 
Many  of  these  responsibilities  include  ensuring  that  various  checks  are  performed. 
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26.  OTD  Responsibilities  Prior  to  Test  Operations 

The  OTD  performs  various  checks  as  the  date  to  begin  test  operations  approaches. 
Checks  include  the  following: 

•  Appropriately  trained  personnel  will  be  available  to  operate  and  maintain  the  equipment 

•  The  equipment  to  be  evaluated  will  be  installed  and  checked  out 

•  Operator  and  maintenance  manuals,  the  Integrated  Logistic  Support  Plan/Acquisition 
Logistic  Support  Plan  (ILSP/ALSP),  NATP,  and  other  necessary  documentation  will  be 
available  from  the  DA 

•  Instrumentation  will  be  available  and  in  working  order 

•  Targets,  simulators,  electronic  warfare  services,  etc.,  will  be  available 

•  Participants  have  received  and  understand  test  plans  and  LOIs 

•  RTD&E  support  services  are  on  track 

•  Contingency  plans  are  available  for  the  unexpected 

•  Arrangements  have  been  make  for  pretest  briefings,  including  arrangements  for 
additional  briefers  if  needed 

•  Rehearsals  of  test  operations  are  scheduled  if  appropriate 

•  Pre-faulted  modules  will  be  available  for  an  M-DEMO  if  necessary 

Immediately  prior  to  the  start  of  test  operations  the  OTD  ensures  the  following: 

•  All  hands  know  what  they  are  supposed  to  do 

•  The  equipment  to  be  evaluated  is  in  working  order 

•  Equipment  necessary  to  the  test  scenario  and  instrumentation  equipment  is  in  working 
order 

•  Personnel  to  activate  and  deactivate  data  recorders,  and  backup  data  takers  are  in  place 

•  Time  synchronization  and  communications  have  been  established  as  necessary 

•  Contingency  plans  have  been  discussed  with  appropriate  personnel 

27.  OTD  Responsibilities  During  Test  Operations 

During  test  operations  the  OTD  ensures  that: 

•  Tests  are  conducted  per  the  test  plan  and  the  LOI;  any  deviations  are  noted,  their  impact 
assessed,  and  necessary  corrective  action  taken;  and  contingency  plans  are  implemented 
as  necessary 

•  All  hands  are  briefed  on  the  test  plan  and  understand  their  roles 
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•  COMOPTEVFOR  is  advised  of  any  potential  issues  that  could  result  in  a  COI  being 
unresolved  unsatisfactorily 
Reports  are  generated  as  specified  in  the  test  plan 
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APPENDIX  C:  FULL  LIST  OF  BRAINSTORMING  IDEAS 


1 .  Incorporate  backwards  compatibility  in  standards  for  OT 

2.  DT  exit  criteria  to  include  OT  pass  criteria 

3.  Move  towards  open  architecture 

4.  Use  actual  shipboard  hardware  and  software  at  the  end  of  DT 

5.  Use  a  hull  just  for  OT 

6.  Rotate  active  user  representatives  in  the  PM  office 

7.  Specific  SETR  for  interoperability 

8.  Pre-testing  training  for  operators/users 

9.  Standardize  test  requirements 

10.  Interoperability  simulation  standards 

1 1 .  Validate  other  systems  meet  interoperability  requirement  prior  to  OT 

12.  Include  testing  interoperability  criteria  into  Mission  Readiness  Assessment  (MRA) 
reviews 

13.  Use  of  mission  threads  for  interoperability 

14.  Inform  the  OPTEVFOR  about  the  system  requirements  early  on 

15.  Create  a  new  color  of  money  for  testing 

16.  Consider  Interoperability  at  all  milestone  reviews 

17.  Improve  the  default  requirements  list 

18.  Involve  interoperability  Tech  Warrant  Holder  early 

19.  Financial  incentive  for  PM 

20.  Delete  requirements  before  test  that  PM  knows  the  system  doesn’t  meet 

2 1 .  Interoperability  Readiness  Levels 

22.  Ensure  that  all  of  the  interfaces  are  synchronized 

23.  CONOPS  and  DRMs  must  be  fed  into  test  plans 

24.  Validate  mission  critical  sequence  with  the  user 

25.  Longer  terms  for  PMs 

26.  Don’t  test  Interoperability 

27.  *  Salary  penalties  for  PM 

28.  *  Write  tests  that  match  requirements 

29.  *  Allow  PMs  to  self-certify  OT  passes 

30.  *  Do  not  finalize  test  requirements  procedures  until  system  is  ready  to  test 

31.  *  No  OT  until  PM  says  all  requirements  are  met 

32.  *  PM  review  of  COI  prior  to  OT 

33.  *  Increase  schedule  allotted 

34.  *  Combine  all  DoD  into  a  single  joint  service 

35.  *  Increase  funding 

36.  *  Increase  test  and  requirements  funding 

37.  *  Write  better  requirements 

38.  *  Develop  simulation  of  your  system  for  future  systems 

39.  *  Create  a  full  test  bed  that  does  not  require  a  ship 

40.  *  Consider  legacy  systems 

41.  *  Use  better  standards 
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42.  *  Write  more  complete  MIL-STDs 

43.  *  Stop  improving  technologies 

44.  *  Involve  users  in  DT  test  plans 

45.  *  Better  DT  testing 

46.  *  Have  users  involved  in  DT 

47.  *  PM  accountability  over  the  lifecycle  of  the  system 

48.  *  Revise  milestone  C  revision  gate  requirements  to  include  interoperability 

49.  Regression  testing  after  fixing  a  problem  found  in  DT 

50.  Improve  communication  between  interoperability  personnel 

5 1 .  Ensure  the  future  programs  have  a  minimal  technology  standard/  phase  out  legacy 
equipment 

52.  Need  to  work  with  stakeholder  to  ensure  realistic  requirements 

53.  Include  industry  or  academia  assessment  of  requirements 

54.  Harsher  simulations  (have  a  list  of  possible  failures  to  test  against) 

55.  Have  higher  fidelity  simulators  so  more  realistic  tests  can  be  run 

56.  Use  real  test  data  from  previous  systems  to  simulate  events 

57.  Reevaluate  the  passing  criteria  for  interoperability 

58.  Robot  testers,  incorporate  automated  test  evaluations 

59.  Define  in  writing  the  grading  criteria,  trying  to  overcome  that  subject  nature  of  Human 
evaluation 

60.  Compare  evaluations  from  several  different  people,  and  provide  an  integrated  score  or 
assessment  form  those  evaluators 

6 1 .  Set  Technical  competence  requirement  for  government  representative  who  play  the 
liaison  role  for  the  project 

62.  Use  risk  matrix  to  assess  issues/problems 

63.  Force  PMs  to  captain  the  ship 


*  Indicates  ideas  that  were  considered  redundant  or  impractical  and  were  not  evaluated  for 
effectiveness  and  cost 
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APPENDIX  D:  TABLE  TOP  EXERCISE  DOCUMENTS 


This  appendix  contains  documents  used  in  support  for  the  ACIH  TTX.  The  ACIH  is  a 
fictitious  system  created  by  the  authors.  All  documents  in  this  appendix  were  created  by  the 
authors.  Note  that  several  of  these  documents  are  excerpts  from  larger  documents;  therefore  the 
formatting  of  table  and  figure  numbers  is  not  consistent  throughout  this  appendix. 

ACIH  EXECUTIVE  SUMMARY 

Equipment  for  United  States  soldiers  must  offer  advantages  and  capabilities  far  greater 
than  that  of  the  enemy  in  the  battlefield.  A  fully-integrated  battle  suit,  consisting  of  a  combat 
helmet  and  body  suit,  provides  the  vital  protection  that  is  needed  for  the  soldiers  to  successfully 
complete  their  mission.  The  combat  helmet  is  the  most  complex  piece  of  gear  in  the  battle  suit 
offering  the  protection  required  for  any  battlefield  situation.  The  helmet  also  provides  audible 
and  visual  communication  between  headquarters  and  ground  troops  and  improves  and  enhances 
the  solder’s  visual  awareness  using  night-vision  and  thermal  imaging  devices.  The  objective  of 
the  system  architecture  design  is  to  create  a  helmet  that  can  integrate  audio  and  visual 
communications,  intelligence  information,  display  information,  and  provide  head  protection. 

One  of  the  most  important  aspects  of  the  helmet  is  to  convey  communication  between  the 
squadron  team  and  the  command  headquarters.  The  helmet  will  be  connected  to  communication 
devices  that  transmit  and  receive  audio,  video,  GPS,  and  navigation  communication.  It  will  have 
internal  speakers  and  a  microphone  to  relay  audible  signals  for  communication  between  the 
squads  and  command  center.  The  helmet  will  also  be  equipped  with  an  external  video  camera 
that  allows  the  ground  troops  and  headquarters  to  view  the  ongoing  operation  as  it  is  in  progress. 
The  helmet  will  be  a  major  part  of  the  human  interface  that  improves  and  enhances  situational 
awareness  along  with  providing  head  protection  against  blunt  and  ballistic  impact  for  the 
warfighter. 
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DOCUMENTS  USED  FOR  ASR  TTX 


Includes: 

•  ACIH  AoA 

•  ACIH  OV-1 

•  ACIH  Risk  Assessment/Management  Plan 

•  ACIH  Requirements 

•  ACIH  Notional  Schedule 

•  ACIH  Technology  Development  Acquisition  Phase  Plan 
1.  ACIH  AoA 

The  AoA  was  performed  via  a  Concept  Generation  process.  A  morphological  matrix  was 
used  as  the  tool  used  for  concept  generation  for  the  ACIH  components.  The  matrix  includes 
several  alternatives  for  each  characteristic  or  property  needed  to  perform  functions  of  the  ACIH. 
Characteristics  for  components  arose  from  analyzing  the  leaf  node  functions.  In  general  leaf 
nodes  are  functions  that  were  simulated.  Many  variations  and  combinations  of  solutions  can 
then  be  pursued. 

Once  emergent  characteristics  were  identified,  they  were  listed  in  the  matrix.  Next  the 
team  brainstonned  to  come  up  with  possible  alternatives.  A  wide  array  of  alternatives  was 
considered  and  listed  in  morphological  matrix  to  obtain  a  broad  range  of  possible  solutions.  The 
resultant  morphological  matrix  is  listed  in  Table  3. 
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Table  3  Morphological  Matrix  -  ACIH  Functional  Allocation 


Characteristic 

Alternatives 

Navigation 

GPS 

Bathymetric 

Encryption/Decryption 

KG-95 

Software 

based 

Element  H/W 
based 

Transmission 

Tin  can 

Video,  audio 

Audio,  video, 
data 

Audio 

Receiver 

Audio 

Audio,  video 

Audio,  video, 
data 

Tin  can 

Image  Capturing 

Normal 

capture 

Image  snap 
shot 

Capture  with 

zoom 

capability 

Draw  on 
paper 

Night  Vision 

Gen  I 

Gen  II 

Gen  III 

Gen  IV 

Thermal  Vision 

Infrared 

camera 

Uncooled 

thermal 

imaging 

device 

Cryogenically 
thermal 
imaging  device 

Processor 

Intel 

AMD 

Video  Display 

CRT  visor 

Projection 
onto  visor 

OLED  visor 

Handheld 

monitor 

Head  Protection 

Full  face 
helmet 

Open  face 
helmet 

Flip-up  helmet 

Concept  Selection 

Concept  selection  continued  from  concept  generation  and  was  perfonned  by  using  Pugh 
Matrices  as  a  tool.  The  alternatives  from  the  morphological  matrix  were  considered  and 
analyzed  in  a  Pugh  matrix.  Specific  components  were  selected  and  these  components  were 
created  in  CORE  and  mapped  from  functions.  Table  5  shows  the  Pugh  matrix  for  the  Audio 
Video  Transmission. 
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Table  5  Pugh  Matrix  Audio  Video  Transmission 


Digital  Light  Weight  Audio  Video 
Transmitter 

Criteria 

TX-MOD3 

CVT-1400 

DT-200 

2.4  GHz  Digital 

Weight 

F) 

+ 

- 

+ 

Size 

A 

- 

- 

+ 

Rugged 

T 

S 

S 

S 

Consumption 

U 

- 

+ 

- 

Digital 

M 

- 

s 

s 

Sum  of  Positives 

2 

1 

2 

Sum  of  Sames 

1 

2 

2 

Sum  of  Negatives 

2 

1 

1 

Encryption/Decryption 

The  communications  between  the  soldiers  is  accomplished  through  wireless  connection 
and  therefore  the  security  of  the  communication  becomes  an  important  concern  for  the  soldiers. 
As  such,  all  communications  include  audio  and  video  must  be  encrypted  for  transmit  and 
decrypted  at  the  receiver  end.  Again,  the  weight,  size,  and  power  consumption  of  the  component 
other  than  the  component  function  are  important  factors  when  integrating  it  with  the  ACIH. 
While  the  secure  communication  devices  offer  compact,  light  weight,  secure,  user-friendly, 
portable  voice  and  data  communication,  it  is  still  an  external  component  that  adds  a  burden  to  the 
real  estate  of  the  ACIH.  An  alternative  to  the  solution  is  to  perfonn  the  functions  of  encryption 
and  decryption  with  the  encryption  algorithm  One-Time  pads.  Using  this  algorithm,  the  process 
of  encrypting/decrypting  data  requires  very  little  computation,  and  the  generation  of  the  random 
pads  can  be  accomplished  on  the  processor.  The  software  encryption/decryption  is  ideal  for  this 
type  of  weight,  size,  and  power  consumption  limitation.  Table  6  shows  the  Pugh  evaluation 
matrix  for  the  encryption  and  decryption  component. 
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Table  6  Pugh  Matrix  Data  Encryption/Decryption 


Digital  Light  Weight  Audio  Video 
Transmitter 

Criteria 

TX-MOD3 

CVT-1400 

DT-200 

2.4  GHz  Digital 

Weight 

F) 

+ 

- 

+ 

Size 

A 

- 

- 

+ 

Rugged 

T 

S 

S 

S 

Consumption 

U 

- 

+ 

- 

Digital 

M 

- 

s 

s 

Sum  of  Positives 

2 

1 

2 

Sum  of  Sames 

1 

2 

2 

Sum  of  Negatives 

2 

1 

1 

Night  Time  Vision  and  Thermal  Imaging 

The  Generation  I,  II,  III,  and  IV  Night  Time  Vision  technology  are  the  key  alternatives 
for  the  Provide  Night  Time  Vision  function.  The  infrared  camera,  uncooled  thennal  imaging 
device,  and  cryogenically  thennal  imaging  device  are  the  key  alternatives  for  the  Provide 
Thermal  Vision  function.  The  top  alternatives  are  detennined  using  the  Pugh  matrix  in  the 
concept  selection  phase.  The  Pugh  matrix  provides  the  tool  to  evaluate  and  compare  the  physical 
architecture  components  that  best  suited  to  carry  out  the  functions.  The  following  Pugh  matrices 
of  alternative  design  concepts  for  the  night  time  and  thermal  vision  are  evaluated  and  compared 
using  vital  criteria  for  the  stakeholder,  in  Tables  7  and  8. 
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Table  7  Pugh  Night  Time  Vision 


Night  Time  Vision 

Criteria 

GEN  I 

GEN  II 

GEN  III 

GEN  IV 

Image  Resolution 

- 

D  + 

Light  Amplification 

- 

- 

A 

+ 

MTTF 

- 

1 

u 

- 

Signal  to  Noise  Ratio 

- 

- 

M 

+ 

Sum  of  Positives 

0 

0  3 

Sum  of  Sames 

0 

0 

0 

Sum  of  Negatives 

4 

4 

1 

Operating  Temperature  D  + 


Sensitivity  -  ^  + 

Resolution  U 

-  M  + 

Sum  of  Positives  0  3 

Sum  of  Sames  0  0 

Sum  of  Negatives  3  0 


The  key  criteria  for  the  night  time  vision  concepts  are  image  resolution,  light 
amplification,  mean  time  to  failure  (MTTF),  and  signal  to  noise  ratio.  The  key  criteria  for  the 
thermal  imaging  concepts  are  operating  temperature,  sensitivity,  and  resolution.  The  design 
concepts  are  evaluated  and  ranked  with  respect  to  the  design  criteria  to  determine  the  preferred 
concepts. 
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The  trade  off  analysis  for  the  night  time  vision  using  the  Pugh  matrix  shows  the 
Generation  III  night  time  vision  technology  as  preferred  alternative  with  the  best  design  concept. 
The  Generation  I  and  II  do  not  ranked  highly  in  the  scoring  matrix.  The  Generation  IV 
technology  is  rejected  due  to  its  drawback  in  the  mean  time  to  failure  (MTTF).  In  addition,  the 
matrix  for  the  thermal  imaging  depicts  the  cryogenically  thermal  imaging  device  as  the  preferred 
design  concept.  The  device  surpasses  the  datum  concept  of  uncooled  thermal  imaging  device 
when  compared  using  the  specified  design  criteria.  The  cryogenically  thermal  imaging  device  is 
the  best  design  concept  that  meets  the  stakeholder  requirements. 

The  selected  design  concepts  are  transferred  into  CORE  as  Component  elements  for  the 
system  architecture  development.  The  Generation  III  night  vision  technology  and  the 
cryogenically  thennal  imaging  device  are  the  Component  elements  that  perform  the  Provide 
Night  Time  Vision  function  and  the  Provide  Thennal  Vision  function.  The  2.4GHZ  Digital 
Video  and  Audio  transmitter  performs  video  and  audio  transmission.  The  Encryption/decryption 
is  based  on  software  algorithm  such  as  One-Time  Pad.  These  elements  are  represented  and 
related  to  the  remaining  the  architecture  model  as  shown  below. 

Navigation 

For  navigation  we  had  4  choices,  Paper/Digital  Maps  &  Compass,  GPS,  Bathymetric  and 
Celestial  Navigation. 

Using  Paper/Digital  Maps  &  Compass  consist  of  a  sailor  carrying  a  map  of  the  area  and 
using  a  compass  to  finds  his/her  location.  The  soldier  would  need  to  be  contacting  Head  Quarters 
(HQ)  or  the  rest  of  team  to  report  his/her  location.  The  size  of  the  map  would  depend  on  the  size 
of  the  area.  The  accuracy,  the  update  rate  and  speed  for  the  soldier  to  calculate  the  position  and 
transmit  the  info  would  depend  on  the  map,  the  compass,  the  surrounding  area,  the  present 
situation  (hostile  and  none  hostile  environment),  and  the  experience  of  the  soldier. 

Using  a  GPS  devise  would  require  no  soldier  interface.  The  GPS  would  be  part  of  the 
helmet.  The  size  of  the  devise  would  be  smaller  than  any  map.  The  devise  updates  at  about  1 
record  per  second,  and  using  a  processor  and  a  transmitter  it  would  automatically  sends  its 
position  to  HQ  or  the  rest  of  the  team.  The  accuracy  for  a  GPS  Standard  Positioning  is  <15 
meters,  95%  typical. 
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Bathymetry  is  use  to  measure  the  depths  of  large  bodies  of  water,  like  oceans,  seas,  and 
lakes.  It  has  more  use  for  underwater  assignments.  It  would  not  work  on  our  dessert  hostage 
scenario. 

The  last  option  we  got  is  using  celestial  navigation.  It  uses  the  sun,  moon,  stars,  or  planets 
to  find  your  way  around.  It  would  require  a  map,  a  Sextant  and  tables.  This  is  a  complex  and 
involved  process  that  involves  a  fair  amount  of  calculating,  correcting,  referring  to  tables, 
knowledge  of  the  heavens  and  the  Earth,  as  well  as  a  lot  of  common  sense. 

Using  a  Pugh  evaluation  matrix,  we  selected  for  criteria:  weight,  size,  reliability, 
accuracy,  update  rate,  portability,  speed  to  process/  transmit  data  and  require  power.  GPS  score 
7  (+),  1  (same)  and  1  (-).  Bathymetric  and  Celestial  navigation  didn’t  have  a  positive  score.  See 
Table  9. 


Table  9  Pugh  Matrix  Navigation 


Navigation 

2 

Criteria  Q 

3 

S3 

C/3 

C/3 

GPS 

Bathymetric 

Celestial  Nav 

Weight 

+ 

- 

- 

Size 

+ 

S 

S 

Reliability  ^ 

S 

S 

S 

Accuracy  j 

+ 

- 

- 

Update  Rate  u 

+ 

- 

- 

Portability  M 

+ 

- 

s 

Speed  to 

4- 

Process/Transmit  data 

Sum  of  Positives 

6 

0 

0 

Sum  of  Sames 

1 

4 

4 

Sum  of  Negatives 

1 

5 

4 
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Receiver 

Based  on  the  Pugh  selection  matrix  the  Tin  Can  option  is  not  a  viable  choice.  Neither  is 
the  audio  option,  although  it  could  be  an  improved  audio.  Both  the  audio/video  combination  and 
the  audio/video/data  combination  should  be  pursued  further.  Considering  stakeholder  needs 
outlined  in  the  Design  Reference  Mission,  the  best  alternative  to  pursue  at  this  time  is  the 
audio/video/data  receiver.  See  Table  10  for  the  Pugh  matrix. 


Table  10  Pugh  Matrix  Receiver 


Receiver 

Criteria 

Audio 

> 

c 

& 

o’ 

<: 

ol 

a> 

o 

Audio/Video/Data 

Tin  Can 

Weight 

D 

+ 

+ 

- 

Size 

A 

s 

- 

- 

1 

Reliability 

u 

+ 

+ 

- 

Accuracy 

M 

+ 

+ 

- 

Sum  of  Positives 

3 

3 

0 

Sum  of  Sames 

1 

0 

0 

Sum  of  Negatives 

0 

1 

4 
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Image  Capturing,  Processing,  and  Video  Display 


Table  11  Pugh  Matrix  Image  Capturing 


Image  Capturing 


Criteria 

Contour  HD 

GoPro  Hero 

Olympus  D-595 

Weight 

- 

S 

Size 

S 

D 

- 

Reliability 

+ 

A 

+ 

Accuracy 

+ 

I 

u 

- 

Update  Rate 

s 

M 

+ 

Portability 

- 

+ 

Sum  of  Positives 

2 

3 

Sum  of  Sames 

2 

1 

Sum  of  Negatives 

2 

2 

Concerning  image  capturing:  there  were  three  options:  normal  capture  (Contour  HD),  capture 
with  zoom  (GoPro  Hero  Wide),  and  image  snap  shot  (Olympus  CA  Media  D-595).  We  are  comparing  the 
cameras  as  a  whole  for  parts,  which  will  be  later  broken  down  into  the  helmet.  Comparing  the  cost, 
Contour  HD  is  the  most  expensive  option  as  it  provides  HD  video.  The  products’  durability  are  all 
relatively  equal,  with  the  Olympus  camera  being  less  so.  Concerning  resolution,  the  Contour  HD  and  the 
Olympus  both  can  capture  high  resolution  images.  Concerning  the  frames  per  second,  the  Contour  HD 
has  the  highest  capability  shooting  either  60  fps  at  SD  or  30  fps  at  HD,  while  the  GoPro  shoot  only  30  fps 
SD.  The  Olympus  is  not  a  camcorder,  and  greatly  losses  in  this  area.  The  weight  of  the  cameras  is  the 
same,  while  the  Olympus  is  a  bit  lighter  due  to  its  ability.  The  GoPro  and  has  a  fixed  zoom  ability,  while 
the  Contour  HD  does  not.  The  Contour  does  shoot  higher  resolution  images  which  can  be  optically 
zoomed.  The  Olympus  has  the  greatest  zoom  ability  with  a  3x  optical  zoom  with  high  resolution.  Based 
on  the  Pugh  chart,  the  Olympus  has  the  most  benefits;  however  it  comes  at  a  great  cost,  it  actually  is  not 
sending  video,  but  slowly  sending  images.  The  Contour  HD  on  the  other  hand  seems  to  offer  the  best 
performance  out  of  the  three  options.  See  Figure  1 1  for  the  Pugh  matrix. 
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The  image  capturing  is  built  in  the  video  component  system,  and  is  implemented  by  the  soldiers  on  the 
field.  The  image  capture  will  stabilize  the  lens  and  capture  the  image  to  be  used. 

Processor 


Table  12  Pugh  Matrix  Processing 


Processor 

Criteria 

Intel  17 

Intel  Core  2  Quad 

AMD  Phenom 

Cost 

- 

D 

+ 

Clock  frequency 

+ 

A 

S 

Power  Usage 

+ 

T 

S 

L2  Cache 

+ 

U 

s 

Performance 

+ 

M 

- 

Sum  of  Positives 

4 

1 

Sum  of  Sames 

0 

3 

Sum  of  Negatives 

1 

1 

Table  12  shows  the  Pugh  matrix  for  the  processor.  There  are  two  main  producers  of  processor 
chips:  Intel  and  AMD.  The  options  chosen  were  high  end  processors,  while  also  including  the  newest 
high-end  processor  (Intel  i7).  Even  though  these  processors  are  our  options,  a  more  suitable  processor 
may  be  decided  later  after  processor  requirements  get  further  refined.  The  three  options  for  processors 
are  the  Intel  i7,  Intel  Core  2  Quad,  and  the  AMD  Phenom  11  X4.  These  were  compared  using  five 
characteristics:  cost,  clock  frequency,  power  usage,  L2  Cache,  and  performance.  The  Intel  i7  is  the 
newest  processor  and  because  of  that  it  is  the  most  costly  of  the  bunch.  However,  this  gives  the  processor 
the  best  performance  compared  to  the  other  two.  Comparing  the  other  two  processors,  they  match  with 
each  other  very  closely.  However,  the  AMD  has  been  known  for  being  somewhat  cheaper  than  the  Intel 
brand,  making  their  Phenom  processor  slightly  cheaper  than  the  Intel  Core  2  Quad.  However,  when 
comparing  performance  Intel  Core  2  Quad  has  a  slight  edge  compared  to  AMD’s  Phenom.  Based  on  the 
Pugh  Chart,  the  new  Intel  i7  provides  the  most  positives  between  the  options. 
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The  processor  is  built  from  the  custom  SW  to  be  run  on  the  system  and  will  be  implemented  by 
the  soldiers  in  the  field.  The  processor  will  perform  a  large  array  of  tasks  including:  encryption  of  data, 
decryption  of  data,  compiling  the  navigational  map,  displaying  enhanced  vision,  and  displaying  video. 


Table  13  Pugh  Matrix  Video  Display 


Display 


Criteria 

OLED 

LCD 

Projection 

Cost 

- 

D  S 

Transparency 

+ 

A 

Brightness 

+ 

T  + 

Resolution 

S 

u  s 

Size  (Volume) 

+ 

M 

Sum  of  Positives 

3 

1 

Sum  of  Sames 

1 

2 

Sum  of  Negatives 

1 

2 

There  are  three  options  for  the  video  display  (Table  13):  an  OLED  screen  that  can  be  attached  to 
a  visor,  a  projector  to  place  images  onto  the  visor,  and  a  removable  LCD  screen.  These  options  were 
compared  with  five  characteristics:  cost,  see  through,  brightness,  resolution,  and  size.  The  most 
expensive  option  is  the  OLED  since  it  is  technology  that  is  relatively  new.  The  projection  method  and 
LCD  method  are  more  experienced  technology  with  a  similar  price  point.  “Transparency”  is  a 
characteristic  important  for  a  visor  display.  The  OLED  can  display  a  clear  picture  with  a  flexible  screen 
as  thin  as  paper,  which  is  transparent.  The  projection  method  will  project  its  image  onto  the  visor; 
however,  this  may  reduce  the  transparency  of  the  visor.  The  LCD  is  not  transparent  at  all  and  will  need  to 
be  removable.  The  brightness  of  the  LCD  and  OLED  are  equivalent,  while  the  projection  may  suffer 
visibility  issues  during  daylight.  All  three  options  have  the  same  resolution.  Volume  wise,  the  OLED 
will  take  the  least  amount  of  space  in  the  helmet  compared  to  the  other  methods.  The  LCD  will  take  up 
the  most  volume.  Based  on  the  Pugh  Chart,  the  OLED  provides  the  best  performance. 

The  OLED  display  will  be  built  in  the  helmet  and  implemented  by  the  soldiers  in  the  field.  This 
component  is  the  main  interface  for  the  soldier,  displaying  the  navigational  map,  video,  and  enhanced 
vision. 
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Provide  Head  Protection 

A  Pugh  matrix  was  put  together  to  take  a  look  at  alternatives  to  the  concept  design  of  the 
helmet.  Criteria  were  provided  by  the  stake  holders  as  to  the  needs  the  helmet  is  to  meet  during 
mission  use.  The  criteria  provided  included  strength,  comfort,  heat  protection,  impact 
cushioning,  shrapnel  protection  and  volume.  Three  helmet  concepts  were  compared;  full  faced 
helmet,  open  faced  helmet  and  flip  face  helmet.  All  three  had  positives  and  negatives.  The  best 
alternative  to  select  is  the  full  face  helmet.  Compared  with  the  other  helmets,  the  full  face 
helmet  meets  95%  of  the  user's  needs.  See  Table  14. 


Table  14  Pugh  Matrix  Head  Protection 


Helmet 

Criteria 

Full  Face 

Flip  Face 

Open  Faced 

Strength 

S 

S 

Comfort 

- 

- 

Heat  Protection 

+ 

1  + 

Impact 

+ 

T  + 

Cushioning 

u 

Shrapnel 

+ 

M  s 

Protection 

Volume 

+ 

s 

Sum  of  Positives 

4 

2 

Sum  of  Sames 

1 

3 

Sum  of 

1 

1 

Negatives 

1 

1 

Selected  Components 

The  trade  off  analysis  for  components  was  perfonned  using  the  Pugh  matrices  of 
alternative  design  concepts.  The  selected  design  concepts  are  transferred  into  CORE  as 
Component  elements  for  the  system  architecture  development.  Figure  12  shows  a  hierarchical 
diagram  with  the  allocated  physical  components  based  on  the  Pugh  matrices  evaluation. 
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Figure  12  ACIH  Component  Selection  and  Allocation  Hierarchy  Diagram 
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2. 


ACIH  OV-1 


The  OV-1  shows  a  hostage  rescue  scenario  where  the  Advanced  Combat  Integrated 
Helmet  provides  the  soldiers  with  increased  situational  awareness,  showing  them  a  navigational 
map  and  allowing  them  to  know  where  their  other  team  members  and  enemies  are  located.  The 
helmet  allows  for  constant  encrypted  video  and  audio  communication  between  the  soldiers  and 
the  headquarters. 
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3.  ACIH  RISK  ASSESSMENT/MANAGEMENT  PLAN 


The  risk  assessment  and  associated  risk  management/mitigation  plan  includes  the 
evolving  external  environment. 

ASR  review  assesses  the  alternative  systems  that  have  been  evaluated  during  Concept 
Refinement,  primarily  through  the  AoA,  and  ensures  that  the  preferred  system  is  cost  effective, 
affordable,  operationally  effective  and  suitable,  and  can  be  developed  to  provide  a  timely 
solution  to  a  need  at  an  acceptable  level  of  risk.  Of  critical  importance  to  this  review  is  the 
understanding  of  available  system  concepts  to  meet  the  requirements  from  the  ICD  or  the  CDD, 
as  well  as  the  affordability,  operational  effectiveness,  technologies,  and  adaptability  to  the 
evolving  external  environment,  risk  inherent  in  each  alternative  concept. 

One  of  the  artifacts  of  this  review  is  to  provide  a  comprehensive  assessment  on  the 
relative  risks  associated  with  including  Commercial  Off-The  Shelf  (COTS)  or  Non- 
Developmental  Items  (NDI)  as  opposed  to  a  new  design,  with  emphasis  on  host  platform 
environmental  design,  diagnostic  information  integration,  dependence  on  other  government 
programs  and  maintenance  concept  compatibility. 

All  the  components  including  hardware  and  software  used  in  the  ACIH  shall  be  designed 
to  provide  encrypted  and  reliable  communication  serves  as  Operational  Awareness  to  the  special 
operation  soldiers  throughout  the  entire  operation.  The  chosen  components  shall  be  capable  of 
operate  in  a  harsh  environment.  With  fewer  dollars  available  for  research,  test  and  evaluation, 
and  procurement  of  new  systems,  an  important  advantage  of  many  COTS  and/or  NDI 
acquisitions  is  the  reduced  acquisition  cycle  time.  This  reduction  results  primarily  from 
decreased  in  design  and  engineering  time  due  to  the  fact  that  the  COTS  components  have  already 
been  tested  and  have  gone  through  general  acceptance  of  the  product  in  the  commercial 
marketplace  or  in  a  previous  military  application. 

However,  there  are  unique  risks  associated  with  COTS-based  /  NDI  acquisitions.  Most  of 
the  COTS  components  used  commercially  are  not  designed  for  used  in  the  military  operational 
environments.  Failing  to  realize  the  impacts  of  using  COTS  components  can  add  to  system  cost, 
schedule  and  performance  risks.  Therefore,  this  review  must  address  some  of  the  risk  factors 
and  the  mitigations  in  order  to  keep  the  system  development  under  cost  and  low  risk. 
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COTS-based  /  NDI  Component  Risk  Factors: 

1 .  Rapid  and  asynchronous  changes 

2.  Different  obsolescence  impacts 

3.  Proprietary  data. 

4.  Higher  life  cycle  costs 

5.  Multiple  configurations 

6.  Commercial  standards 

7.  Time-limited  manufacturer  support 

8.  Infonnation  security  susceptibility 


Mitigations: 

1 .  Involve  COTS  -  knowledgeable  individuals  in  all  analytical  processes 
Benefit  -  Facilitates  the  application  of  COTS  mitigation  strategies  and  informed 
decision  making 

2.  Involve  users  early  and  throughout  the  program  life  cycle  to  identify  and  resolve 
COTS-related  constraints 

Benefit  -  Reduces  chances  of  surfacing  user  acceptance  issues  late  in  system 
development  and  deployment 

3.  Perfonn  continuous  COTS  product  market  research 

Benefit  -  Allows  product  team  to  project  and  plan  for  changes  in  technology, 
product  configurations  and  obsolescence-related  issues 

4.  Integrate  market  research  results  with  field  data  and  new  requirements 
Benefit  -  Optimizes  and  prioritizes  cost,  schedule  and  perfonnance  factors 
between  obsolescence-driven  system  changes  and  system  upgrades 

5.  Develop  and  maintain  flexible  perfonnance  requirements  suited  to  the  use  of 
COTS  products 

Benefit  -  Allows  for  appropriate  level  of  specified  function  description  and  the 
inclusion  of  COTS  technical  performance  factors 

6.  Institute  and  maintain  ongoing  COTS  product  testing  capability 
Benefit  -  Allows  project  to  assess  new  COTS  products/technologies  for 
specification  compliance,  form/fit/  function  compatibility  and  standards 
conformance 

7.  Integrate  COTS-based  technology  evolution  planning  within  the  Integrated 
Program  Plan 

Benefit  -  Provides  centralized  planning  that  captures  system  evolution  strategy, 
obsolescence  projections  and  risk  mitigation  decisions 

8.  Emphasize  strong  and  COTS  relevant  configuration  management  practices 
Benefit  -  Reduces  the  possibility  of  untested  COTS  product  changes  affecting 
system  performance  and  supports  multiple  system  configurations 

9.  Use  a  COTS-experienced  systems  integration  agent 

Benefit  -  Facilitates  acquisition,  development,  deployment  and  support  activities 
with  proven  COTS  capable  personnel  and  services 

10.  Leverage  the  commercial  infrastructure  wherever  feasible 
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Benefit  -  Prevents  costly  duplication  of  already  existing  COTS  product  support 
infrastructure 

1 1 .  Ensure  the  chosen  hardware  and  software  conform  to  IA  compliance 

Benefit  -  Reduces  the  chances  of  unauthorized  access  to  the  system  while 
increasing  inter-operability  among  different  products  that  meet  commercial 
standards. 


ACIH  Operational  Effectiveness  Risk  Factors: 

The  combat  helmet  is  a  vital  piece  of  protective  equipment  hat  is  essential  in  the 
battlefield.  The  helmet  should  provide  basic  head  protection  as  well  as  advance  capabilities  in  a 
mission  allowing  an  increase  in  situational  awareness.  Proper  integration  of  audio  and  video 
components  provides  soldiers  critical  communication  capability  in  such  the  team  can 
strategically  plan  their  operation  to  counteract  the  enemy  and  carry  out  their  mission.  Risk 
factors  associate  with  operational  effectiveness  are  as  follow: 

1 .  Communications  (audio  and  video)  intercept  by  enemy 

2.  Communications  signal  being  jammed 

3.  Power  consumption  limitation 

4.  Transmit  and  receive  distance 

5.  Helmet  falls  in  enemy’s  hand 

Mitigations: 

1 .  All  communications  (audio  and  video)  are  encrypted.  The  communications  are 
encrypted  with  one  time  secret  key.  The  secret  key  is  used  to  encrypt  and  decrypt 
audio  and  video  signal.  The  strength  of  the  secret  key  must  exceed  the  life  time 
of  the  mission 

2.  Anti-jamming  algorithm  prevent  jamming  signal  interrupting  communications 

3.  The  battery  shall  have  proper  capacity  to  provide  power  to  audio  and  video 
components  throughout  the  entire  operation. 

4.  The  transmitter  and  receiver  shall  be  capable  of  operate  effectively  within 
communications  distance  between  soldiers. 

5.  The  one  time  secret  key  shall  be  changed  for  each  operation 
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4.  ACIH  REQUIREMENTS 


NEEDS 

In  order  to  meet  the  needs  outlined  in  the  executive  summary,  the  helmet  must: 

•  Be  well  ventilated  and  comfortable  to  wear  in  lengthy  operation. 

•  Properly  seat  on  the  head  without  obstructing  the  user’s  view. 

•  Absorb  and  reduce  energy  from  blunt  and  ballistics  impact  to  reduce  the  risk  of 
head  injury. 

•  Provide  the  capability  enabling  the  soldiers  to  see  and  operate  in  an  environment 
with  or  without  visible  illumination. 

•  Allow  the  soldiers  to  carry  out  their  search,  identification,  and  monitoring  of 
enemy  activities  during  all  times  of  day  as  they  prepare  and  organize  their 
operation. 

•  Allow  the  soldiers  to  exchange  situational  infonnation  and  video  with  one  another 
as  well  as  their  command  headquarter. 


The  Projected  Operational  Environment  (POE)  of  the  Advanced  Combat  Integrated 
Helmet  (ACIH)  is  designed  to  be  used  in  the  desert  area  such  as  Iraq  and  Afghanistan.  The  ACIH 
is  mostly  used  by  Special  Forces  in  the  operation  of  a  hostage  rescue  mission.  Desert  warfare  is 
highly  vulnerable  to  foreign  annies  that  are  not  familiar  or  experienced  with  the  area.  Knowing 
how  to  navigate  in  the  desert  is  the  desert  fighter's  best  advantage.  The  ACIH  will  have 
communication  equipment  (transmit  and  receive),  and  GPS  and  navigation,  to  provide  better 
maneuvering  and  situational  awareness  during  the  rescue  operation  in  the  desert  environment. 


DRM  Example 

A  group  of  anned  hostiles  take  defensive  positions  in  an  urban  city.  Hostages  are  taken 
to  deter  engagements  on  their  location.  The  objective  is  to  retreat  or  eliminate  soldiers.  The 
mission  is  to  infiltrate  the  building  undetected,  detect  the  enemy,  capture  or  eliminate  hostile 
enemies,  and  secure  the  hostages. 


Mission  Success  Requirement 

The  ACIH  provides  advanced  combat  operation  architecture  to  ensure  special  combat 
operation  soldiers  can  effectively  fight  in  a  diverse  range  of  operational  environments  to  carry 
out  their  mission.  Its  ultimate  goal  is  to  accomplish  Enhance  User  to  Perform  Rescue  Mission. 
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The  mission  is  defined  as  a  success  if  the  following  conditions  are  true: 

•  Enemy  forces  and  hostages  are  identified 

•  Enemy  and  hostages  activities  are  monitored 

•  Enemy  and  hostages  activities  are  relayed  to  the  squadron  and  command  headquarter 

•  Operation  is  carried  out  at  any  time  of  day 

•  Enemies  are  captured 

•  Hostages  are  rescued 

•  Soldiers  and  hostages  survive  the  rescue  operation 

ACIH  Requirements: 

•  The  Advanced  Combat  Integrated  Helmet  (ACIH)  requirements  shall  provide  and 
maintain  audio  and  video  communications  throughout  entire  hostage  rescue  operational. 

•  The  system  shall  provide  Operational  Awareness  including  the  ability  to  transmit  and 
receive  user  and  team  location,  as  well  as  receive  intelligence  from  headquarters. 

•  The  system  shall  enable  to  user  to  detect  an  enemy  at  a  minimum  distance  of  400  meters 
in  any  environment,  lighting,  or  condition. 

•  The  ACIH  shall  protect  the  user  from  blunt  force  trauma,  sharp  objects,  explosions,  and 
bullet  fragments.  The  ACIH  will  protect  the  user  from  injury,  death,  and  illness. 

•  The  ACIH  helmet  shall  share  encrypted  data  to  protect  users  and  the  mission. 

•  The  ACIH  helmet  shall  process  the  data  and  output  the  required  infonnation  to  the  user. 

•  The  system  shall  have  a  MTBF  of  no  less  than  3900  hours. 

•  The  system  shall  operate  after  impact  from  blunt  forces. 

•  The  system  shall  operate  without  hindering  users’  performance. 
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5.  ACIH  NOTIONAL  SCHEDULE 
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6.  ACIH  TECHNOLOGY  DEVELOPMENT  ACQUISTION  PHASE  PLAN 

The  purpose  of  this  phase  is  to  reduce  technology  risk  and  to  detennine  the  appropriate 
set  of  technologies  to  be  integrated  into  the  full  system.  This  will  ensure  that  the  proposed 
technology  solution  is  affordable,  militarily  useful,  and  based  on  mature  technology. 

Entrance  Criteria 

A  successful  completion  of  AoA  and  proposed  materiel  solution  is  presumed  at  this  stage 
of  the  acquisition  process. 

Phase  Description: 

1 .  The  Technology  Development  Phase  begins  when  the  MDA  has  approved  a  material 
solution  and  the  Technology  Development  Strategy  (TDS). 

2.  The  DoD  component  shall  submit  a  cost  estimate  for  the  proposed  solution(s)  identified 
by  the  AoA. 

3.  Final  Requests  for  Proposals  (RFPs)  for  the  Technology  Development  Phase  shall  not  be 
released  until  the  MDA  has  approved  the  TDS. 

Technology  and  Development  Plan  for  Advanced  Combat  Integrated  Helmet  (ACIH) 
Description  of  Approach 

The  advanced  combat  helmet  is  a  complex  piece  of  gear  that  provides  vital  protection 
needed  for  the  soldiers  to  successfully  complete  their  rescue/hostage  mission.  The  helmet 
provides  audible  and  visual  communication  between  headquarters  and  ground  troops  to  enhance 
situational  and  operational  awareness.  The  real-time  audio  and  video  data  are  processed  and 
transferred  using  the  current  transmitter  and  receiver  technology  to  facilitate  instant 
communication.  The  latest  and  up-to-date  situational  and  operational  information  is  critical  in  a 
successful  engagement  planning  and  operational  maneuver. 

1 .  The  selected  approach  is  preferred  due  to  the  advanced  transmitter  and  receiver  technology 
that  is  widely  available  for  immediate  use.  The  use  of  transmitter  and  receiver  to  exchange 
audio  and  video  data  and  infonnation  is  advantageous  as  the  maturity  of  the  existing 
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technology  has  been  previously  studied  and  verified.  It  is  more  assuring  that  the  transmitter 
and  receiver  will  likely  to  perform  as  designed  when  earlier  studied  and  documented  data  are 
examined  and  considered  during  the  system  design  and  development.  An  acceptable  risk 
level  can  be  detennined  from  the  use  of  commercial  components  to  ensure  that  it  is  suitable 
for  the  overall  system  interoperability.  The  determination  of  suitable  transmitter  and  receiver 
for  the  proposed  system  is  successfully  completed  by  using  Pugh  matrices.  Refer  to  the  AoA 
section  for  the  Pugh  Matrix  for  each  component. 

a.  A  2.4  GHz  audio  transmitter  is  selected  for  the  transmission  of  audio  and  video  data. 
The  transmitter  offers  the  greatest  capability  with  given  criteria. 

b.  A  Contour  HD  camera  provides  the  required  capability  to  capture  image  and  process 
video.  The  camera  offers  HD  video  quality  with  high  resolution.  It  is  able  to  provide 
the  highest  image  capturing  capability  at  30  frames  per  second.  This  feature  is  ideal 
for  the  diverse  operating  environment  of  the  proposed  advanced  helmet  system. 

c.  The  audio/video/data  receiver  provides  the  needed  capability  for  communication 
exchange.  The  selection  fulfills  the  operational  need  of  enhancing  situational 
awareness  in  a  combat  operation. 

d.  OLED  screen  is  the  selected  component  that  will  provide  clear  picture  or  video 
display.  The  OLED  is  a  thin  transparent  flexible  screen  that  can  be  attached  to  a 
helmet  visor  to  view  video  data. 

2.  A  software  interface  will  be  used  to  integrate  individual  component  into  a  full  system. 
The  software  application  will  be  used  to  manage,  control,  and  maintain  the 
interoperability  of  microphone,  headphone,  and  video  display.  Equipment  readiness  is 
monitored  to  detennine  their  operating  condition.  Data  and  information  exchange 
between  components  is  also  processed  and  handled  by  the  software  program.  The  data  is 
encrypted  and  decompressed  prior  to  transmission  then  decrypted  and  uncompressed 
upon  receipt.  This  ensures  that  the  information  is  securely  transferred  to  intended 
recipients.  The  One-Time  pads  is  the  selected  component  that  provides  a  secure  data 
transfer.  . 
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The  data  encryption  and  decryption  will  be  used  in  conjunction  with  a  processor  to 
process  audio  and  video  information.  Intel  17  processor  provides  the  best  processing 
power  and  speed  for  such  infonnation  exchange.  Refer  to  the  AoA  section  for  the  Pugh 
Matrix  for  encryption/decryption  and  the  processor 

3.  A  Technology  Development  Strategy  will  be  prepared  for  additional  reviews. 

4.  A  preliminary  acquisition  strategy,  including  cost,  schedule,  and  performance  goals  for  the 
total  research  and  development  program  will  be  provided  during  the  review. 

5.  Final  Requests  for  Proposals  (RFPs)  for  the  Technology  Development  Phase  will  be 
submitted. 
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DOCUMENTS  USED  FOR  OTRR  TTX 


Includes: 

•  ACIH  Operational  Requirements  Document 

•  ACIH  Operational  Test  Plan  and  Scenarios 

•  ACIH  Training  Plan 

•  ACIH  DT&E  Test  Data 

•  ACIH  DT&E  Test  Report 

•  ACIH  Notional  Integrated  Logistics  Support  Plan 
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1. 


ACIH  OPERATIONAL  REQUIREMENTS  DOCUMENT 


The  ACIH  Requirements  artifact  for  the  ASR  was  used  as  the  Operational  Requirements 
Document  (ORD). 


Operational  Requirements  Document 


Advance  Combat  Integrated  Helmet 


October  27,  2010 
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OPERATIONAL  REQUIREMENTS  DOCUMENT 


1.  General  Description  of  Operational  Capability. 

Special  operation  soldiers  in  combat  typically  operate  in  a  diverse  range  of 
operational  environments  and  are  vulnerable  to  injury  threats  placing  demands  on  the 
soldiers’  protective  system  to  provide  consistent  perfonnance  throughout  a  range  of 
situations  and  threats.  The  combat  helmet  is  a  vital  piece  of  protective  equipment  that  is 
essential  in  the  battlefield.  Stakeholders  seek  a  comfortable  helmet  that  provides  basic 
head  protection  as  well  as  advanced  capabilities  in  a  mission  allowing  an  increase  in 
situational  awareness  as  such  that  the  team  can  strategically  plan  their  operation  to 
counteract  the  enemy  and  carry  out  their  mission  of  hostage  liberation. 

1.1.  Mission  Need. 

The  equipment  for  United  States  soldiers  must  offer  advantages  and  capabilities 
far  greater  than  that  of  the  enemy  in  the  battlefield.  A  fully-integrated  battle  suit, 
consisting  of  a  combat  helmet  and  body  suit,  provides  the  vital  protection  that  is  needed 
for  the  soldiers  to  successfully  complete  their  mission.  The  combat  helmet  is  the  most 
complex  piece  of  gear  in  the  battle  suit  offering  the  protection  required  for  any  battlefield 
situation.  In  addition  of  providing  audible  and  visual  communication  between 
headquarters  and  ground  troops,  the  helmet  also  improves  and  enhances  the  solder’s 
visual  awareness  using  night-vision  and  thennal  imaging  devices.  The  objective  of  the 
system  architecture  design  is  to  create  a  helmet  that  can  integrate  audio  and  visual 
communications,  intelligence  information,  display  information,  and  provide  head 
protection. 

1.2.  Overall  Mission  Area  Description. 

The  ACIH  is  designed  to  be  used  during  the  day  or  night  in  the  desert  area  such 
as  Iraq  and  Afghanistan.  The  ACIH  is  mostly  used  by  Special  Forces  in  the  operation  of 
a  hostage  rescue  mission.  Desert  warfare  is  highly  vulnerable  to  foreign  annies  that  are 
not  familiar  or  experienced  with  the  area.  Knowing  how  to  navigate  in  the  desert  is  the 
soldier’s  best  advantage.  The  ACIH  will  have  communication  equipment  (transmit  and 


195 


receive),  and  GPS  and  navigation,  to  provide  better  maneuvering  and  situational 
awareness  during  the  rescue  operation  in  the  desert  environment. 

1.3.  Descriptions  of  the  Proposed  System. 

One  of  the  most  important  aspects  of  the  helmet  is  to  provide  reliable  and 
encrypted  communications  between  the  squadron  team  and  the  command  headquarters. 
The  helmet  shall  be  capable  of  transmit  and  receive  audio,  video,  GPS,  and  navigation 
communication  Audio  communication  between  the  squads  and  command  center  shall  be 
accomplished  without  interruption  causes  by  harsh  operational  environment.  The  helmet 
shall  also  be  providing  real-time  video  stream  on  the  battle  ground  in  such  that  allows  the 
ground  troops  and  headquarters  to  view  the  ongoing  operation  as  it  is  in  progress.  The 
helmet  will  be  a  major  part  of  the  human  interface  that  improves  and  enhances  situational 
awareness  along  with  providing  head  protection  against  blunt  and  ballistic  impact  for  the 
warfighter.  It  is  the  most  Advanced  Combat  Integrated  Helmet  (ACIH)  that  gives  special 
operation  soldiers  ability  to  strategically  plan  their  operation  and  carry  out  their  mission 
with  minimum  casualty. 

1.4.  Supporting  Analysis. 

1.5.  Mission  the  Proposed  System  Will  Accomplish. 

1.6.  Operational  and  Support  Concept. 

1.6.1.  Concept  of  Operations. 
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The  OV-1  shows  a  hostage  rescue  scenario  where  the  Advanced  Combat 
Integrated  Helmet  (ACIH)  provides  the  soldiers  with  increased  situational  awareness, 
showing  them  a  navigational  map  and  allowing  them  to  know  where  their  other  team 
members  and  enemies  are  located.  The  helmet  allows  for  constant  encrypted  video  and 
audio  communication  between  the  soldiers  and  the  headquarters. 

1.6.2.  Support  Concept. 

1.7.  Acquisition  Approach. 


2.  Threat. 

The  threat  is  an  armed  hostile  group  considered  to  be  trained  in  firearms  and  explosives.  The 
hostiles  are  operating  in  a  desert  urban  environment.  Threat  is  assumed  to  have  no  surveillance 
equipment  installed. 

Threat  Properties: 

Average  Size:  8  hostiles 

Weapons:  Kalashnikov  AK-47,  Unknown  Explosive  Devices 
Potential  Hostages:  2-3 

The  threat’s  courses  of  action  (COA)  are  defined  as  follows: 
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Most  Likely  COA:  A  disorganized  armed  group  of  hostiles  retreat  to  a  civilian  location,  taking  a 
defensive  position  within  a  building.  The  hostiles  will  silently  wait  till  the  situation  has  died 
down  before  retreating  to  an  enemy  encampment.  The  hostiles  will  shoot  at  soldiers  on  sight  if 
they  suspect  their  location  is  compromised. 

Most  Dangerous  COA:  An  aimed  team  infiltrates  a  civilian  location  and  reinforces  the  area  with 
small  scale  explosives.  The  hostiles  take  civilian  hostages  to  deter  possible  engagements.  The 
hostile’s  goal  is  to  eliminate  as  many  soldiers  as  possible,  shooting  any  soldier  on  sight.  They 
have  no  regard  for  their  own  lives. 

3.  Existing  System  Shortfalls. 


Operational  Shortfalls.  The  ACIH  is  designed  to  combine  advance  combat 
technology  to  meet  current  and  projected  requirements.  It  is  designed  to  provide  comfortable 
head  protection  against  blunt  and  ballistics  impact  and  allow  the  soldiers  to  operate  in  low  to  no 
illumination  environment.  The  helmet  is  designed  to  increase  situational  awareness  by  providing 
real  time  communication  between  the  ground  troops  and  headquarters.  The  immediate 
communication  can  provide  valuable  command  and  control  and  decision  support  in  the  search, 
identification,  and  monitoring  of  enemy. 


4.  Capabilities  Required. 

4.1.  Operational  Performance  Parameters. 
Conditions: 


Environment 

Requirement 

Developed  System 

Projected  Operation 

Temperature 

-30  F  to  1 10  F 

-50  F  to  130  F 

60  F  to  110  F 

Time  of  Day 

Night  and  Day 

Night  and  Day 

Night 

Environment 

Desert,  Urban,  Snow 

Desert,  Urban,  Snow 

Desert 

4.2.  ORD  Key  Performance  Parameters  (KPPs). 

•  Operator  shall  receive  continuous  geographical/navigational  satellite  data  with  a  lag  of 
less  than  3  seconds. 

•  The  encrypted  video  data  shall  be  transmitted  and  received  no  greater  than  1  second. 

•  The  encrypted  audio  data  shall  be  transmitted  no  greater  than  1  second  and  received  no 
greater  than  0.25  seconds. 

•  The  navigational  data  shall  be  transmitted  in  less  than  0.5  second. 
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•  The  ACIH  high  grade  absorbent  padded  lining  shall  absorb  blunt  force  within  6 
milliseconds. 

•  The  ACIH  Kevlar  shall  protect  the  user  from  shrapnel  traveling  at  340  meters/second. 

•  The  ACIH  shall  protect  the  user  from  heat  and  fire  generated  from  a  grenade  explosion  at 
4  seconds. 

4.3.  System  Performance. 

4.3.1.  Mission  Scenarios. 

A  group  of  anned  hostiles  take  defensive  positions  in  an  urban  city.  Hostages  are  taken 
to  deter  engagements  on  their  location.  The  objective  is  to  retreat  or  eliminate  soldiers.  The 
mission  is  to  infiltrate  the  building  undetected,  detect  the  enemy,  capture  or  eliminate  hostile 
enemies,  and  secure  the  hostages. 

4.3.2.  System  Performance  Parameters. 

4.3.2. 1.  The  audio  communication  equipment  such  as  headsets  shall  reduce  noise 
exposure  while  delivering  clearer  audio  and  comfortable  fit.  High  noise  levels  in 
surrounding  environment  can  affect  situational  awareness.  Misunderstood  commands  and 
repeated  instructions  can  increase  operational  risk  and  reduce  mission  effectiveness.  The 
headsets  offer  active  and  passive  technologies  which  together  provide  full-spectrum  noise 
reduction  with  comfortable,  lightweight  designs  that  can  be  worn  continuously  during 
long  missions. 

4.3.2.2.  With  the  capability  of  the  secure  video  communication,  the  soldiers 
during  the  hostage  rescue  operation  can  gain  insight  of  the  hostile  environment  via  real¬ 
time  video  communications.  The  video  communication  should  leverage  satellite 
communication  that  provides  a  secure  wireless  video  communications  infrastructure 
between  command  center  and  the  ground  soldiers.  The  video  communication  provides 
the  soldiers  visibility  and  intelligence  by  feeding  the  soldiers  imagery  resulted  of 
reconnaissance  and  coordinated  mission  planning.  It  also  provides  real-time  streaming 
video  of  the  operation  to  command  center  to  assess  the  operation  developments. 

4.3.2.3.  The  ACIH  shall  use  GPS  to  coordinate,  record  soldier  and  team  location 
during  a  hostage  rescue  operation.  This  includes  determining  distance,  direction, 
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location,  elevation/altitude,  route,  and  data  for  navigation  aids,  orientation  and  rate  of 
movement. 

4.3.2.4.  The  ACIH  shall  enable  the  soldier  to  detect  enemies  at  a  minimum 
distance  of  400  meters  in  any  environment,  lighting,  or  condition. 

4.3.2.5.  The  system  shall  amplifies  light  to  allow  the  soldier  to  see  at  dark.  This 
allows  the  soldier  to  see  in  levels  of  light  that  approach  total  darkness.  The  device  shall 
use  a  system  no  older  than  Generation  3  (GEN  III). 

4.3.2.6.  The  systems  shall  enable  soldier  to  see  enemy  heat  signatures. 

4.3.2.7.  The  ACIH  will  protect  the  soldier’s  head  from  injury,  illness,  and  death  as 
a  result  of  blunt  force  trauma,  sharp  objects  (knives),  and  explosions. 

4.3.3.  Information  Exchange  Requirements. 

4.3.4.  Interoperability. 

System  interoperability  involved  the  ability  for  the  systems  of  system  to  provide 
and  accept  services  from  each  other  to  enable  them  to  operate  effectively  together.  For 
ACIH  system,  audio,  video,  and  navigational  data  must  be  exchanged  via  the  transmitter 
and  receiver  to  achieve  the  desired  services.  The  data  from  the  microphone  and  speaker 
is  processed  with  software  and  delivered  to  the  user  after  completion. 

4.3.5.  Logistics  and  Readiness. 

4.3.6.  Other  System  Characteristics. 

5.  System  Support. 

5.1.  Maintenance. 

5.2.  Supply. 

5.3.  Training. 

5.4.  Transportation  and  Facilities. 

6.  Force  Structure. 

7.  Schedule. 
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8.  System  Affordability. 
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2.  ACIH  OPERATIONAL  TEST  PLAN  AND  SCENARIOS 


ADVANCE  COMBAT 
INTEGRATED  HELMET  (ACIH) 

OPERATIONAL  TEST  & 
EVALUATION  (OT&E) 
TEST  PLAN 


27  October  2010 


PREPARED  BY 

COMMANDER,  OPERATIONAL  TEST  and 
EVALUATION  FORCE  (COMOPTEVFOR) 
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Section  1 

Introduction  to  the  Project 


1.0  Introduction 

The  purpose  of  this  Operational  Test  (OT)  event  is  to  assess  the  Advanced  Combat  Integrated 
Helmet  (ACIH)  by  identifying  system  enhancements  and  significant  areas  of  risk  to  the  programs 
successful  completion  of  OT&E.  OT  ACIH  will  be  conducted  from  15-19  November  2010  at 
the  U.S.  Anny  Yuma  Proving  Ground  Test  Center,  Yuma,  AZ. 

The  level  of  risk  is  associated  with  the  successful  resolution  of  Critical  Operational  Issues  (COI) 
during  OT  ACIH.  This  risk  is  based  upon  assessment  of  thresholds,  program  documentation, 
program  plans,  and  Subject  Matter  Expert  (SME)  analysis. 

1.1  System  Description 

The  ACIH  is  intended  to  integrate  audio  and  visual  communications,  intelligence  information, 
display  infonnation,  and  provide  head  protection.  The  ACIH  will  convey  communications 
between  a  squadron  and  command  headquarters.  It  will  be  connected  to  communication  devices 
that  transmit  and  receive  audio,  video,  Global  Positioning  System  (GPS),  and  navigation 
communication.  It  will  have  internal  speakers  and  a  microphone  to  relay  audible  signals  for 
communication  between  the  squadrons  and  command  centers.  The  helmet  is  also  equipped  with 
an  external  video  camera  that  allows  the  ground  troops  and  headquarters  to  view  the  ongoing 
operation  as  it  is  in  progress. 

The  ACIH  consists  of  a  2.4  GHz  Digital  Light  Weight  Audio  Video  Transmitter,  One-Time  Pads 
Data  Encryption  /  Decryption,  GEN  III  Night  Vision,  Cryogenically  Thermal  Vision,  GPS 
Navigation,  Audio/Video/Data  Receiver,  Contour  HD  (Image  Capturing,  Processing,  and  Video 
Display),  Intel  17  Processor,  OLED  Display,  and  a  full  face  helmet. 
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Section  2 

Administrative  Information 


2.0  General 

General  responsibilities  of  activities  involved  in  the  testing  are  provided  in  this  section,  as  well 
as  appropriate  points  of  contact.  Continuing  close  liaison  is  essential  to  timely  and  successful 
prosecution  of  this  event. 

2.1  Responsibilities 

COMPTEVFOR 

1 .  Provide  changes  to  this  test  plan. 

2.  Conduct  briefings  for  all  participating  units,  including  Operations  Security  (OPSEC) 
requirements  and  procedures. 

3.  Supervise  data  collection,  participate  in  test  analysis,  and  publish  final  report. 
Program  Executive  Office  Integrated  Warfare  System  (PEO  IWS) 

1 .  Furnish  required  material  and  technical  support. 

2.  Provide  for  installation  and  maintenance  of  project  equipment. 

3.  Provide  funding  for  data  collection,  reduction,  and  analysis  support. 

4.  Provide  for  and  coordinate  targets  and  range  support. 

5.  Provide  for  appropriate  safety  certifications. 

6.  Certify  system  readiness  for  OT  ACIH  per  SECNAVINST  5000. 2C. 

2.2  Points  of  Contact 


Points  of  contact  are  provided  in  table  2-1. 


Table  2-1  i 

Points  of  Contact 

Name/Rank 

Title  (Code) 

Address 

Commercial 

CDR  John  Doe 

Operational  Test 
Coordinator  (Code  72) 

Commander, 
Operational  Test  and 
Evaluation  Force 

7970  Diven  St 

Norfolk,  VA  23505 

757-222-2222 

2.3  Visitor  Control 
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SECNAV’S  policy  regarding  visitor  observance  of  Operational  Testing  (OT)  is  strict.  This  is  to 
preclude  any  perception  of  a  lack  of  objectivity  in  the  T&E  process  or  any  perception  of  outside 
influence  on  the  OT  unit  and/or  OTD.  Therefore,  observers  will  not  normally  be  permitted  in  the 
test  area  during  OT.  During  testing,  visit  authorization  will  be  controlled  by  Program  Executive 
Office  Integrated  Warfare  Systems  (PEO  IWS)  and  granted  only  for  valid  requirements  or  for 
technical  assistance. 

2.4  Disclosure  Policy 

Test  Data.  Factual  OT  data  will,  as  expeditiously  as  possible,  be  released/shared  with  the 
program  office.  The  logistics  of  release/sharing  of  data  will  not  interfere  with  the  conduct  or 
evaluation  of  any  OT.  Factual  data  does  not  include  information  based  on  consensus  or  opinion, 
such  as  operator  or  maintainer  surveys.  Such  information  is  subjective  and  part  of  the  evaluation 
process,  and  will  not  be  made  available  prior  to  the  release  of  the  final  report.  DOT&E  access  to 
test  data  will  be  per  applicable  sections  of  Title  10. 

Proprietary  Information.  Requests  for  access  to  proprietary  infonnation  will  be  referred  to  the 
proprietor  agency  for  disposition.  Proprietary  infonnation  will  not  e  disclosed  by 
COMOPTEVFOR.  Information  collected  by  the  OTD  in  the  fonn  of  survey  sheets  (user  and  test 
team  feedback,  comments,  opinions,  and  conjecture  of  system  performance)  during  OT 
constitutes  proprietary  information  of  COMOPTEVFOR.  This  includes  infonnation  gathered 
from  questionnaires  and  interviews.  Such  infonnation  will  be  labeled:  “FOR  OFFICIAL  USE 
ONLY  -  NOT  RELEASABLE  OUTSIDE  OF  COMOPTEVFOR. 

Deviations  from  the  test  plan.  The  OTD  is  authorized  to  deviate  from  this  test  plan  as  the 
operational  situation  and  good  judgment  dictates,  keeping  COMOPTEVFOR  advised. 
COMOPTEVFOR  will  advise  DOT&E  of  deviations  from  this  test  plan. 

Release  of  information  to  the  press  or  other  agencies.  Prior  to  formal  issue  of  the  final  report, 
no  test  data  will  be  released.  Once  the  report  is  issued  by  COMOPTEVFOR,  the  CNO  will 
release  data  per  existing  policy.  Media  requests  to  observe  OT  will  be  refened  to  Chief  of 
Infonnation  (CHINFO)  in  Washington,  DC.  Requests  for  other  than  OT&E  infonnation  will  be 
referred  to  CHINFO  for  coordination  with  CNO  and  COMNAVSEASYSCOM. 
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Section  3 

Scope  of  the  Evaluation 


3.1  Critical  Operational  Issue  (COIs):  COIs  for  OT  ACIH: 

COIs 

Tests 

Maintain  Communications 

E-l 

Provide  Operational  Awareness 

E-2 

Enable  User  Vision 

E-3 

Reliability 

S-l 

Maintainability 

S-2 

Availability 

S-3 

Interoperability 

S-4 

Training 

S-5 

3.2  Evaluation  Criteria 

CNO  provided  the  required  Measures  of  Effectiveness  (MOE),  Measures  of  Suitability,  and 
Critical  Technical  Parameters  (CTP).  Table  3-1  is  the  Operational  Activities  for  the  ACIH  and 
MOEs.  Table  3-2  is  the  Critical  Technical  Parameters.  Table  3-3  is  the  Operational  Activities 
that  the  components  of  the  ACIH  must  meet.  COMOPTEVFOR  will  consider  all  of  these  in  the 
assessment  of  the  ACIH.  Data  will  be  collected  and  analyzed  to  obtain  a  characterization  of  the 
ACIH  and  to  detennine  improvement  trends  throughout  the  testing  cycle. 


Table  3-1:  Operational  Activities  for  the  ACIH 


Operational  Activity 

Description 

MOE 

Identify  Target  of 

Operation 

Friend/Foe 

Identification 

•  User  shall  receive  a  constantly  updated 
map  with  a  lag  no  greater  than  3  seconds. 

Transmit  Operational 

Infonnation 

Provide  situational 

and  operational 

information  in 

planning  for 

engagement 

•  The  video  data  shall  transmit  encrypted 
data  in  no  greater  than  1  second 

•  The  audio  data  shall  transmit  encrypted 
data  in  no  greater  than  1  second 

•  The  navigational  data  shall  transmit 
encrypted  data  in  no  greater  than  0.5 
seconds 
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Receive  Operational 

Infonnation 

Obtain  situational 

and  mission 

information  to 

facilitate  operational 

maneuver 

•  The  audio  shall  reach  the  user  with  a  lag 
no  greater  than  0.25  seconds. 

•  The  video  shall  reach  the  user  with  a  lag 
no  greater  than  1  second. 

•  The  user  shall  receive  a  constantly 
updated  map  with  a  lag  no  greater  than  3 
seconds. 

Request  Command  & 

Decision  Support 

Request  for  support 

from  command 

center  for  operation 

planning  or  decision 

making  for  tactical 

maneuver  and 

engagement 

•  The  video  data  shall  transmit  encrypted 
data  in  no  greater  than  1  second 

•  The  audio  data  shall  transmit  encrypted 
data  in  no  greater  than  1  second 

Carry  Out  Rescue 

Operation 

Operational 

maneuver  and 

engagement  to 

detain  enemy  and 

rescue  hostages 

•  The  ACIH  high  grade  absorbent  padded 
lining  shall  absorb  blunt  force  within  6 
milliseconds 

•  The  ACIH  Kevlar  shall  protect  the  user 
from  shrapnel  traveling  at  340 
meters/second 

•  The  ACIH  shall  protect  the  user  from 
heat  and  fire  generated  from  a  grenade 
explosion  at  4  seconds 

•  The  user  shall  receive  a  constantly 
updated  map  with  a  lag  no  greater  than  3 
seconds 

_ Table  3-2:  ACIH  Critical  Technical  Parameters _ 

The  Advanced  Combat  Integrated  Helmet  (ACIH)  requirements  shall  provide  and  maintain  audio 
and  video  communications  throughout  entire  hostage  rescue  operational. 

The  system  shall  provide  Operational  Awareness  including  the  ability  to  transmit  and  receive 
user  and  team  location,  as  well  as  receive  intelligence  from  headquarters. 

The  ACIH  helmet  shall  share  encrypted  data  to  protect  users  and  the  mission. 

The  ACIH  helmet  shall  process  the  data  and  output  the  required  information  to  the  user. 

The  system  shall  have  a  MTBF  of  no  less  than  3900  hours. 


Table  3-3:  Operational  Activities  that  the  components  of  the  ACIH  must  meet 


209 


Component  Function 


Operational  Activity 

a 

d 

_o 

_o 

"d 

%— » 
d 

%— * 
d 

d 

d 

U 

<D 

J-H 

<D 

"d 

Oh 

o 

0) 

d 

Oh 

o 

o 

d 

d 

_o 

d 

a 

a  g 

d 

d 

_o 
%— * 

o 

O  o 

d 

C/0 

<D 

» 

<D 

bD 

d 

H 

d 

<L>  _ 

Oh  g 

0-2 

O  g; 

d 

Jh 

0) 

o  « 

U  o 

o 

b 

c3 

U 

>> 

%— » 

a 

0> 

*d 

M 

<D  aj 
£ 

0)  fcj 

<D  d 

Pi  h£ 

t/l  O 

3-2 
cr  o 

1)  <D 

Pi  Q 

£  1 

1  O 

H  £ 

2.4  GHz  Digital  Transmit 
Transmitter  Encrypted 
Audio  Data 


Transmit 

Encrypted 

Navigational 

Data 


Transmit 
Encrypted 
Video  Data 


Audio  Output  Audio 

Communication  Communication 
System 


Receive  Audio 
Communication 


Transmit  Audio 
Communication 


Display 

Marked 

Imagery/Map 


Transmit  User 
Location 


Provide  Head 
Protection 


Continuously 
Mark  Team 
Location 


Decrypt 

Incoming 

Audio 


Helmet 


Intel  i7 
Processor 


210 


Component  Function 
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3.3  Testing 

Test  operations  will  exercise  the  ACIH  at  the  Yuma  Proving  Ground.  System  testing  will 
provide  data  for  operational  effectiveness  (E-tests)  and  operational  suitability  (S-tests)  COIs. 
These  are  discussed  further  in  sections  4  and  5. 
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Section  4 

Operational  Effectiveness 

4.1  Scenarios 

Test  scenarios  are  based  on  a  squadron  of  10  Special  Forces  soldiers  using  the  ACIH.  Their 
mission  is  to  rescue  a  group  of  three  hostages  who  are  being  held  captive  by  a  group  of  8 
hostiles.  The  hostiles  are  considered  to  be  trained  in  firearms  and  explosives,  and  are  armed  with 
Kalashnikov  Ak-47s  and  unknown  explosive  devices.  The  hostages  are  being  held  in  a  desert 
urban  environment.  The  Special  Forces  are  to  be  dropped  via  helicopter  and  once  on  the  ground 
they  are  to  communicate  with  each  other  and  Headquarters  via  the  ACIH.  They  will  download 
satellite  imagery  of  the  terrain  and  building  where  the  hostages  are  being  held.  The  team  will 
then  locate  the  building  using  the  GPS  component  of  the  ACIH.  Once  the  building  has  been 
located,  they  will  then  transmit  voice  and  video  imagery  of  the  situation  to  Headquarters 
awaiting  further  orders.  All  data  will  be  encrypted  and  decrypted  per  OPSEC.  Once  the  order  is 
given,  the  team  will  engage  the  hostiles  and  take  the  hostages  to  a  safe  location  using  the  ACIH 
communication  devices  that  transmit  and  receive  audio,  video,  Global  Positioning  System  (GPS), 
and  navigation  communication.  This  mission  will  be  executed  twice,  once  during  the  day  and 
once  during  the  night.  During  the  evening  mission,  the  Special  Forces  will  locate  the  hostiles 
using  night  vision  and  infrared  components  of  the  ACIH.  All  weapons  engagements  will  be 
simulated.  The  ACIH  will  be  operated  continuously  throughout  the  test  period.  A  detailed 
schedule  of  events  will  be  used  during  OT  ACIH  and  will  be  promulgated  by  the  test  conductor 
prior  to  the  test  period. 

4.2  Test  E-l,  Maintain  Communications 

Objective:  The  ACIH,  operating  with  its  integrated  supporting  components  will 
maintain  audio  and  visual  communications  throughout  entire  hostage  rescue  mission. 

1 .  Receive  Audio  Communication  and  Transmit  Audio  Communication. 

2.  Receive  Video  Feed  and  Transmit  Video  Feed. 

Procedure:  ACIH  will  be  assessed  at  each  test  event  during  the  test  period.  Data  will  be 
recorded  via  observer  notes,  OT  test  team  comment  sheets  and  DX. 

Data  Analysis:  Maintaining  of  Communications  will  be  assessed  qualitatively  and 
quantitatively  based  on  OT  test  team  comment  sheets,  DX  analysis,  and  operational  experience. 
The  ACIH  will  be  evaluated  on  whether  or  not  it  can  reduce  noise  exposure  while  delivering 
clearer  audio  providing  full  spectrum  noise  reduction  with  comfortable,  lightweight  design  that 
can  be  worn  continuously  during  long  missions.  ACIH  will  also  be  evaluated  on  being  able  to 
transmit  and  receive  audio/video  between  team  and  headquarters.  The  video  communication 
should  leverage  satellite  communication  that  provides  a  secure  wireless  video  communications 
infrastructure  between  command  center  and  the  ground  soldiers.  The  video  communication 
provides  the  soldiers  visibility  and  intelligence  by  feeding  the  soldiers  imagery  resulted  of 
reconnaissance  and  coordinated  mission  planning. 

4.3  Test  E-2,  Provide  Operational  Awareness 

Objective:  The  ACIH,  operating  with  its  integrated  supporting  components  will 
maintain  operational  awareness  throughout  entire  hostage  rescue  mission. 

1.  Receive  Team  Location 
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2.  Transmit  User  Location 


3.  Receive  Intelligence  Information 

Procedure:  ACIH  will  be  assessed  at  each  test  event  during  the  test  period.  Data  will  be 
recorded  via  observer  notes,  OT  test  team  comment  sheets  and  DX. 

Data  Analysis:  Providing  Operational  Awareness  will  be  assessed  qualitatively  and 
quantitatively  based  on  OT  test  team  comment  sheets,  DX  analysis,  and  operational  experience. 
The  ACIH  will  be  evaluated  on  whether  or  not  it  can  use  GPS  to  coordinate,  record  user  and 
team  location  during  a  hostage  rescue  operation.  This  includes  determining  distance,  direction, 
location,  elevation/altitude,  route,  and  data  for  navigation  aids,  orientation  and  rate  of  movement. 

4.4  Test  E-3,  Enable  User  Vision 

Objective:  The  ACIH,  operating  with  its  integrated  supporting  components  will  enable 
user  vision  to  detect  enemies  at  a  minimum  distance  of  400  meters  in  any  environment,  lighting, 
or  condition. 

1 .  Provide  Night  Time  Vision  (NTV) 

2.  Provide  Thermal  Vision  (TV) 

Procedure:  ACIH  will  be  assessed  at  each  test  event  during  the  test  period.  Data  will  be 
recorded  via  observer  notes,  OT  test  team  comment  sheets  and  DX. 

Data  Analysis:  Enabling  User  Vision  will  be  assessed  qualitatively  and  quantitatively 
based  on  OT  test  team  comment  sheets,  DX  analysis,  and  operational  experience.  While 
operating  in  levels  of  light  approaching  total  darkness,  the  ACIH  shall  amplify  light  to  allow  the 
user  to  see.  When  operating  in  total  darkness  or  at  night,  the  ACIH  shall  allow  the  user  to  see 
heat  signatures  of  the  enemies. 
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Section  5 

Operational  Suitability 

5.1  General 

The  suitability  testing  will  use  data  generated  by  continuous  operation  of  the  ACIH 
throughout  test  operations,  including  effectiveness  tests  described  in  Section  4.  A 
maintainability  demonstration  will  also  be  conducted.  Tests  specifically  designed  to  generate 
maintainability  data  are  described  in  the  following  Suitability  Tests  (S-Tests). 

5.2  Test  S-l,  Reliability 

Objective:  Will  the  ACIH  reliability  support  completion  of  the  mission? 

Procedure:  Reliability  will  be  assessed  continuously  during  the  test  period. 

Maintenance  action  forms  will  be  completed  for  each  failure  or  issue  noted  during  operations, 
and  for  each  preventive  maintenance  action  that  finds  a  failed  part.  Data  will  be  recorded  via 
observer  notes/OT  test  team  comment  sheets,  maintenance  logs,  and  DX. 

Data  Analysis:  Reliability  will  be  assessed  qualitatively  and  quantitatively  based  on 
observer  notes/OT  test  team  comment  sheets,  maintenance  logs,  and  operational  experience. 
Reliability  quantitative  assessment  is  based  on  calculations  for  MTBOMFHw  and  MTBOWFsw: 

MTBOMFhw  is  the  mean  time  between  operational  mission  hardware  failures  occurring 
during  system  operation  and  is  calculated  by: 

MTBOMFhw  =  Total  System  Operating  Time  /  #  of  Operational  Mission  Hardware 
Failures 

Where  an  operational  mission  hardware  failure  is  one  which  causes  the  ACIH  to  fail  its 
mission.  System  operating  time  includes  only  the  time  the  system  is  operating.  It  does  not 
include  standby  time. 

MTBOMFsw  is  an  operational  mission  software  fault.  An  operational  mission  software 
fault  is  an  interruption  of  ACIH  operation  not  directly  attributable  to  hardware,  which  causes  the 
ACIH  to  fail  its  mission. 

MTBOMFsw  =  Total  System  Operating  Time  /  #  of  Operational  Mission  Software  Faults 

5.3  Test  S-2,  Maintainability 

Objective:  Is  the  ACIH  maintainable  by  Special  Forces? 

Procedure:  Maintainability  will  be  assessed  continuously  during  the  test  period. 
Trouble  and/or  maintenance  action  reports  will  be  completed  and  reviewed  as  appropriate.  Data 
will  be  recorded  via  observer  notes/OT  team  comment  sheets,  and  maintenance  logs. 

Data  Analysis:  Maintainability  will  be  assessed  qualitatively  and  quantitatively  based 
on  observer  notes/OT  test  team  comment  sheets,  maintenance  logs,  and  operational  experience. 
Maintainability  quantitative  assessment  is  based  on  calculations  for  MCMTOMFhw  and 
MCMTOMFsw- 

MCMTOMFhw  is  the  average  elapsed  corrective  maintenance  time  needed  to  repair  all 
operational  mission  hardware  failures. 

MCMTOMFhw  =  Total  Elapsed  Time  to  Correct  Operational  Mission  Failures  /  Total  # 
of  Operational  Mission  Failures. 

5.4  Test  S-3,  Availability 

Objective:  Will  the  ACIH  be  available  to  support  completion  of  the  mission? 

Procedure:  All  OT  test  team  comment  sheets,  maintenance  fonns,  and  time  meter 
recordings  from  tests  S-l  and  S-2  will  be  reviewed. 

Data  Analysis:  Ao  is  computed  using  the  formula:  Ao  =  Uptime  /  Uptime  +  Downtime 
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5.5  Test  S-4,  I/O 

Objective:  Will  the  ACIH  be  interoperable  with  the  systems  with  which  it  must 
interface? 

Procedure:  This  test  will  be  conducted  throughout  the  OT  ACIH.  Test  S-4  will  examine 
the  I/O  between  ACIH  and  interfacing  systems. 

Data  Analysis:  I/O  will  be  assessed  qualitatively  based  on  observer  notes/OT  test  team 
comment  sheets,  maintenance  logs,  and  operational  experience.  The  impact  of  any  I/O  issues  on 
overall  mission  accomplishment  identified  during  testing  will  be  assessed. 

5.6  Test  S-5,  Training 

Objective:  Will  the  ACIH  training  support  system  operation  and  maintenance  by  Special 
Forces? 

Procedure:  At  this  point,  only  adequacy  of  the  ACIH  Navy  Training  System  Plan 
(NTSP)  training  programs  will  be  assessed.  Selected  ACIH  training  courses  may  be  audited  by 
the  OTD  to  determine  adherence  to  NTSP  requirements  and  potential  effectiveness  for  training 
Special  Forces. 

Data  Analysis:  The  ACIH  training  program  will  be  qualitatively  assessed  based  on 
operational  experience  and  judgment.  Training  issues  observed  during  the  assessment  will  be 
evaluated  on  the  basis  of  its  impact  on  overall  mission  accomplishment. 
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Section  6 
Reports 

6.1  General 

Reports  required  in  connection  with  this  project  are  described  in  the  following 
paragraphs.  Distribution  should  be  limited  where  indicated. 

6.2  Readiness  Reports 

PEO  IWS  Certification.  PEO  IWS  shall  certify  the  ACIH  readiness  for  OT  ACIH  per 
SECNAVINST  5000.2C 

Unit  Readiness.  Prior  to  commencement  of  testing,  PEO  IWS  will  submit  a  message 
report  to  COMOPTEVFOR  if  the  Yuma  Proving  Ground  test  site  is  not  ready  to  commence 
operations.  This  report  will  include  the  reason  project  operations  cannot  commence  and  any 
expectations  or  reservations  on  the  part  of  PEO  IWS. 

6.3  OT  Commencement  Report 

Upon  commencement  of  OT  ACIH,  the  ACIH  OTD  will  notify  COMOPTEVFOR 
indicating  actual  start  time  (Date-Time-Group  (DTG)  Zulu)  of  testing.  Comments,  particularly 
anticipated  limitations,  may  be  included  in  this  communication. 

6.4  Status  Reports 

Deficiency  Reports 

A  deficiency  recommendation  will  be  submitted  by  quickest  available  means  directly  to 
COMOPTEVFOR  by  the  ACIH  OTD  when  the  project  is  delayed  because  the  equipment  cannot 
be  operated  properly,  the  required  support  is  lacking,  or  there  has  been  prolonged  delay  in 
equipment  delivery.  The  deficiency  recommendation  will  contain  a  summary  of  the  deficiency, 
action  taken,  and  recommended  corrective  action. 

Anomaly  Reports 

An  initial  anomaly  report  will  be  submitted  by  quickest  available  means  directly  to 
COMOPTEVFOR  by  the  ACIH  OTD  when  failures  or  anomalies  occur  that  impact  OT  and 
require  correction,  but  are  not  so  severe  that  a  deficiency  report  is  required.  The  anomaly  report 
will  identify  the  failure  or  anomaly,  its  impact  on  OT  and  overall  system  perfonnance,  and 
recommend  corrective  action. 

Completion  of  Test  Operations  Report 

Upon  completion  of  the  OT  ACIH  data  analysis  process,  the  ACIH  OTD  will  notify 
COMOPTEVFOR  indicating  the  completion  time  (DTG  Zulu)  of  OT  ACIH. 

6.5  Evaluation  Reports 

Analysis  Report 

Data  Analysis  Team  (DAT)  analysis  reporting  will  be  per  reference  x. 

COMOPTEVFOR  Report 

COMOPTEVFOR  will  submit  a  final  evaluation  report  to  CNO  within  90  days  of 
completion  of  project  operations. 
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3.  ACIH  TRAINING  PLAN 


NAVY  TRAINING  PLAN  FOR  THE 
ADVANCED  COMBAT  INTERGRATED  HELMET 

(ACIH) 

OCTOBER  2010 
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EXECUTIVE  SUMMARY 

Advanced  Combat  Integrated  Helmet  (ACIH)  is  a  modem  day  device  that  helps  enhance 
military  capabilities  in  vital  combat  operation.  The  helmet  helps  improve  situational  awareness 
and  operational  readiness  by  providing  real-time  audio  and  video  communication  between  the 
operating  forces  and  headquarter.  The  exchange  of  up-to-date  operational  situations  helps 
enhance  tactical  strategy  development  in  offensive  and  defensive  combat. 
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PART  I  -  TECHNICAL  PROGRAM  DATA 


A.  Title-Nomenclature-Program 

a.  Title  Nomenclature  Acronym:  Advanced  Combat  Integrated  Helmet  -  ACIH 

b.  Program  Element:  XXXX 

B.  Security  Classification:  Security  documents  are  developed  in  accordance  with 
OPNAINST  C55 13.6C  -  Communication  and  Satellite  security  Guidance  dated  7 
DEC  2005 

a.  Audio  communication  transmit/receive  -  Secret 

b.  Video  Communication  transmit/Receive  -  Secret 


C.  NTP  Principals 

a.  Director  of  Naval  Training 

CNO 

b. 

Bureau  of  Naval  Personnel 

BUPERS 

c. 

Commandant  of  the  Marine  Corps 

CMC  (ASM) 

d. 

Principal  Development  Activity 

NAVSEASYSCOM 

e. 

Assistant  Chief  of  Naval  Operation 

CNO 

f. 

Manpower  and  Personnel  Mission 
Sponsor 

CNO 

D.  Operational  Uses 

a.  Purpose:  The  ACIH  consists  of  audio  and  video  assembly  that  provides  real¬ 
time  communication  between  operational  forces  and  headquarter. 

b.  Foreign  Military  sales  and  Other  Source  Procurement 

E.  Technical  and/or  Operational  Evaluation  (TECHEVAL/OPEVAL):  The  training  plan 
developmental  will  be  verified  during  OTRR.  The  plan  will  be  revised  based  on 
inputs/comments  from  the  review.  The  plan  will  be  testing  during  OT. 

F.  Equipment/S  vs  tem/Sub  system  replaced 

a.  Audio  Communication  System: 

i.  Receive/Transmit  devices  - 1  level  maintenance 

b.  Video  Communication  System: 

i.  Transmit/Receive  devices  -  I  level  Maintenance 

ii.  Display  component 

iii.  Night  time  vision 

c.  Processor  -  I  level  maintenance 


221 


G.  Description 

a.  Functional  Description:  The  ACIH  consists  of  audio  and  video  assembly  that 
provides  real-time  communication  between  operational  forces  and 
headquarter.  The  audio  assembly  facilitates  point-to-point  verbal 
communication  as  the  video  assembly  provides  visual  infonnation  that  is 
displayed  on  the  OLED  screen  on  the  face  visor.  The  communication 
exchange  is  achieved  with  the  use  of  digital  transmitter  and  receiver  to 
transfer  encrypted  information.  The  encrypted  data  is  securely  processed  via  a 
software  interface  called  One-Time  Pads.  The  One-Time  Pads  is  a  software 
program  that  is  used  to  encrypt  and  decompress  information  prior  to 
transmission.  It  is  also  used  to  decrypt  and  uncompressing  data  upon 
reception  by  the  receiver.  The  secured  audio  information  is  sent  to  the  user’s 
earpiece  and  the  video  data  is  displayed  onto  the  screen  on  the  helmet. 

b.  Physical  description:  The  physical  and  electrical  characteristics  of  the  ACIH 
are  as  follows: 

i.  Weight 

ii.  Diameter 

iii.  Power  requirement  for  Audio/Video 

iv.  Power  requirement  for  display 

H.  Training  Concepts 

a.  Operational:  The  training  is  to  provide  the  users  with  the  best  practice  for 
ACIH  operation.  The  user  will  gain  in  depth  knowledge  of  the  operation  and 
maintenance  of  the  helmet  to  ensure  that  it  is  properly  used  and  function 
during  the  mission.  The  training  curriculum  provides  operational  concept  so 
the  operator  will  be  able  to  operate  and  execute  the  ACIH  function 
components 

i.  Transmit/Receive  audio  communication 

ii.  Transmit/Receive  video  communication 

iii.  Display  video  infonnation 

b.  Maintenance:  The  ACIH  maintenance  objective  is  to  provide  the  means  to 
restore  the  unserviceable  unit  back  to  serviceable  condition  with  minimum 
down  time.  The  training  curriculum  provides  the  end  user  information  and 
tool  to  diagnose  problem  and  determine  what  level  support  the  unserviceable 
unit  be  sent  to.  Follow  the  initial  diagnose  results,  the  unserviceable  unit  will 
be  sent  to  one  of  the  three  level  maintenance  which  designed  by  the  Naval 
maintenance  principle 
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i.  Organization  level  maintenance 

ii.  Intermediate  level  maintenance 

iii.  Depot  level  maintenance 

I.  Logistics 

J.  Schedules 

K.  Manpower  Requirements 

L.  On-Board  Training 

M.  List  of  Related  Navy  Training  Plans  and  Applicable 


PART  II  -  BILLET  AND  PERSONNEL  REQUIREMENTS 

A.  Billets  Required  for  Operational  and  field  Support 

a.  Field  Logistic  Support 

b.  Training  refresh  support 

B.  Billets  Required  for  Maintenance 

a.  I-Level  Maintenance 

b.  Depot  Maintenance  level 


PART  III  -  Training  Requirement 

A.  Length  of  the  training  program 

a.  Operator:  1.5  week  course 

i.  ACIH  components  familiar 

ii.  Video  communication  Hardware  operational 

iii.  Audio  communication  Hardware  operational 

iv.  Basic  Networking 

v.  Basic  Satellite  communication 

b.  Maintenance:  3  weeks  course 

i.  ACIH  Hardware  familiar 

ii.  Basic  electronics  theory 

iii.  Basic  Networking  theory 

iv.  Troubleshooting  Video  communication  system 

v.  Troubleshooting  Audio  communication  system 

vi.  Troubleshooting  ACIH  main  computer 
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B.  Logistic  Support 

a.  Special  Test  equipment 

i.  Network  analyzer 

ii.  Signal  generator  -  audio/video 

b.  Tool 


PART  IV  -  Points  of  Contact 

A.  NSWC-PHD  Code  XXX 

B.  FTCLANT  Code  XXX 

C.  FTCPACT  Code  XXX 
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4.  ACIH  DT&E  TEST  DATA 


Six  DT&E  Test  Data  Sheets  for  the  ACIH  were  utilized: 


Test  Scenario  Name:  COMPONENTS 
Test  Scenario  Name:  USER  INTERFACE 

Test  Scenario  Name:  HEADQUARTERS  CONNECTION  TO  SATELLITE 
Test  Scenario  Name:  CONNECTION  BETWEEN  HEADQUARTERS  - 
SATELLITE  -  ACIH 

Test  Scenario  Name:  ACIH  CONNECTION  TO  ANOTHER  ACIH 
Test  Scenario  Name:  CONNECTION  BETWEEN  10  ACIH  UNITS 
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TEST  SCENARIO  NAME:  COMPONENTS 


This  test  would  desmonstrate  the  basic  functionality  of  the  ACIH 
Description:  component 


Objective:  Verify  microphone 

Verify  speaker 
Verify  camera 
Verify  Display 
Verify  helmet 
Verify  transmistter 
Verify  receiver 
verify  encryptor 


Pass/Fail:  PASS 

Requirments 

Step 

1  microphone  Sensitivity 

2  microphone  Impedance 

3  Speakers  Impedance 

4  Speakers  Frequency 

5  Speakers  Sensitivity 

6  OLED  Display  transparent 

7  OLED  Display  thinnes 

8  OLED  Display  ambient  temperature 

9  Helmet  protection 

10  Helmet  weight 

11  Processor  clock  speed 

12  Processor  Frequency 

13  Processor  Bus/Core  Ratio 

14  Processor  Cache 

15  Processor  Memory 

16  Processor  VI D 

17  Transmitter  Test  A- Audio 

18  Transmitter  Test  B-  Video 

19  Transmitter  Test  C-  Navigation 

20  Encryptor  Test  A-  Audio 

21  Encryptor  Test  B-  Video 

22  Encryptor  Test  C-  Navigation 

23  Camera  Test  A-  Zoom 

24  Camera  Test  B-  Night  Vision 

25  Camera  Test  C- Thermal  Vision 

26  Camera  Test  D  -  Weight 

27  Camera  Test  F  -  Temperature 

28  Camera  Test  F  -  Voltage 


Description  P/F 

Sensitivity  to  be  54db  P 

Impedance  to  be  1000  Ohm  P 

Impedance  to  be  80  Ohm  at  1  KHz  P 

Frequency  response  20Hz-  22KHz 

Sensitivity  to  be  1 08db  @  1  KHz  P 

P 

P 

Ambient  temperature  to  be  1 0f  to  120f 
protects  against  9mm  FMJ  and  44  Magnum  P 

4  lb  P 

Mim  2.5  GHz  P 

Min  3.0  GHz  P 

21  P 

Min  6  P 

24  GB  P 

Require  ,70V- 1.34V  P 

Transmit  Audio  P 

Transmit  Video  P 

Transmit  Navigation  data  P 

Software  encrypt  and  decrypt  Audio  data  P 

Software  encrypt  and  decrypt  Video  data  P 

Software  encrypt  and  decrpt  Navigation  data  P 

Camera  zoom  mim  400  meters  P 

Spectral  Response  Visible  to  0.90  pm  (IR)  P 

Resolution  320  x  240,  Detection  range  minimum  400m,  P 
Max  weight  700  grams 
Max  temp  1 20f 
Max  Vdc  3.5 


Comment 


The  max  freequency  response  was  19KHz 


The  max  temperature  before  it  started  to  maifuntion  was 
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TEST  SCENARIO  NAME:  USER  INTERFACE 


This  test  demonstrates  the  basic  functionality  of  turning  on  and  off 
the  ACIH  interfaces:  microphone,  speakers,  transmitter,  receiver, 
Description:  night  vision,  thermal  vision  and  display. 


Objective: 

Verify  connectivity  between  the  ACIH  and  headquarters 

Verify  video  capability 

Verify  text  data  is  received  and  sent 

Verify  vision  and  display  capability 

Pass/Fail: 

PASS 

Step 

Requirements 

Description 

P/F 

Comment 

1 

User  can  turn  on  the  audio. 

The  user  turns  on  the  Audio  (speaker). 

P 

2 

User  can  turn  on  the  microphone 

The  user  turns  on  the  microphone. 

P 

.  ■ 

Establish  audio  communication  with 
3  headquarters. 


Headquarters  sends  a  communication  check.  The  user 
replies  and  confirms  communication  is  established. 


P  Communication  had  a  2-3  seconds  delay 


User  can  stop  sending  voice  communication 
while  still  able  to  receive  communications 
4  from  headquarters. 


User  uses  the  interface  mute  application,  while  still  able 
to  receive  instructions  from  headquarters. 


The  user  push  the  mute  button.  The  user 
was  getting  voice  information  from 
headquarters.  User  started  a  conversation 
with  the  Tester,  headquarters  did  not  heard 
the  user  conversation.  BUT  after  45 
seconds  headquarters  started  hearing  the 
conversation.  Contractor  would  investigate 
this  issue. 


Headquarters  requests  video.  User  turns  "on"  video  and 

6  Verify  video  capability  (part  A  -  ON)  requests  a  video  verification  check  from  headquarters.  P 


Headquarters  requests  to  turn  "off  "video.  The  user 
turns  "off"  the  video  and  requests  a  verification  check 

7  Verify  video  capability  (part  B- OFF)  from  headquarters.  P 


User  turns  on  and  off  Display  capability  and  at  the  same 

8  Verify  Display  Capability  (part  A  ON/OFF)  time  gets  verification  checks  from  headquarters.  P 


User  zooms  in/out  onto  an  object  at  400  meters 

9  Verify  Display  Capability  (part  B- zoom)  distance.  P 


Verify  Display  Capability  (part  C  -  Night 
10  Vision) 


User  turns  on/off  Night  Vision  capability  and  gets 
verification  from  headquarters  . 


P 


Verify  Display  Capability  (part  D  -  Thermal 
Vision) 


User  turns  on/off  Thermal  Vision  capability  and  gets 
verification  from  \ 


It  turned  on,  but  after  about  5  seconds  it 
turned  off  by  itself.  Test  was  repeated  and 
the  second  time  it  turn  off  by  it  self  after  1 0 
seconds.  Contractor  would  investigate  this 
issue. 


Verify  Display  Capability  (part  E  -  Location 
12  map  layout) 


User  receives  map  layout  from  headquarters.  User 
turns  on/off  the  map  layout  display  option. 


P 


227 


TEST  SCENARIO  NAME:  Headquarters  connection  to  Satellite 


This  test  demonstrates  the  basic  functionality  of  sending  and 
Description:  receiving  information  between  the  satellite  and  headquarters. 


Objective:  Verify  connectivity  between  the  Satellite  and  headquarters 

Verify  there  is  constant  information  going  back  and  forward 
Verify  video  data  is  received  and  sent 
Verify  text  data  is  received  and  sent 
Verify  voice  data  is  received  and  sent 


Requirement 

Coverage:  ACIH  -  R1 .  ACIH  -  R2,  ACIH  -  R3,  ACIH  -  R5,  ACIH  -  R7 


Pass/Fail:  PASS 

Comment: 

Step  Requirements  Description  P/F  Comment 

Headquarters  establishes  a  connection  with 

1  satellite  Headquarters  sends  PING  to  the  Satellite  P  It  took  3  attempts 


2  Head  quarters  sends  text  data  to  the  Satellite  Headquarters  sends  a  ’text”  word  to  the  satellite  P  The  word  that  was  choose  was  "MISSION" 


Satellite  receives  text  data  and  sends  it 
3  back  to  headquarters 


Satellite  receives  the  "text”  word  "MISSION"  and  sends 

it  back  to  headquarters.  P 


Obtain  .gif  files  from  Satellite 


Headquarters  requests  a  picture  (.gif  file)  from  the 
satellite 


Headquarters  requests  a  picture  (.jpg  file)  from  the 

6  Obtain  .jpg  files  from  Satellite  satellite  P 


Headquarters  sends  a  10  minute  video  clip  and  receives 

7 

Satellite  receives  and  sends  video 

it  back 

P 

The  receiving  clip  had  a  3  second  delay 

8  Satellite  receives  and  sends  voice 


Headquarters  sends  a  5  minute  voice  clip  and  receives 

it  back  P 


This  clip  was  sent  twice.  First  time  the  clip  was  cutout.  The 
second  time  headquarters  got  the  full  clip  with  about  a  1  second 
delay 


Satellite  receives/sends  voice  and  data  at 
9  the  same  time 


Headquarters  sends/receives  a  5  minute  video  and 

voice  P 


Headquarters  sends/receives  encrypted  data 
10  to  the  satellite  for  24  consecutive  hours. 


The  video  clip  still  has  a  3  seconds  delay.  Connection  was  lost 
Headquarters  sends/receives  voice,  text,  video  data  on  for  2  times  for  a  period  of  1  minute  the  first  time  and  45  seconds 

encryption  mode.  The  test  run  for  24  consecutive  hours.  P  the  second  time;  during  24hrs  of  testing 
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TEST  SCENARIO  NAME:  Connection  between  Headquarters  -  Satellite  -  ACIH 


Description: 

Objective: 


Requirement 

Coverage: 

Pass/Fail: 

Comment: 


This  test  demonstrates  the  basic  functionality  of  sending  and 
receiving  information  between  the  satellite  and  the  ACIH 

Verify  connectivity  between  the  Headquarters  and  ACIH 
Verify  there  is  constant  information  going  back  and  forward 
Verify  video  data  is  received  and  sent 
Verify  text  data  is  received  and  sent 
Verify  voice  data  is  received  and  sent 


ACIH  -  R1 ,  ACIH  -  R2,  ACIH  -  R3,  ACIH  -  R5,  ACIH-6,  ACIH  -  R7 
PASS 

(artifact  =  object,  person  or  target) 


Step  Requirements 


Description 


P/F  Comment 


Headquarters  establishes  connection  with 

1  ACIH  via  Satellite.  Headquarters  sends  PING  to  ACIH  via  Satellite  P 

2  Headquarters  requests  ACIH  position  Headquarters  request  GPS  data  from  ACIH  P 

On  the  first  three  attempts  it  didn't  pass.  Contractor  re¬ 
calibrate  the  ACIH  camera,  try  it  the  4th  time  and  it  didn't 

3  Headquarters  obtains  video  data  from  ACIH  Headquarters  requests  video  data  from  ACIH  P  work.  Contractor  change  the  camera  and  it  work  on  the 


ACIH  receives  GPS  data  from  other  artifacts  Headquarters  sends  GPS  information  from  one  artifact 

4  around  his  position  around  the  ACIH  unit.  P 


6 

ACIH  receives  GPS  data  from  multiple 
artifacts  around  his  position 

Headquarters  sends  GPS  information  from  5  artifacts 
around  the  ACIH  unit. 

F 

The  ACIH  was  only  able  to  receive  GPS  data  from  4 
artifacts.  Contractor  would  keep  working  on  this  issue 

7 

ACIH  sends/receives  voice  data  to  and  from 
headquarters 

Headquarters  receives/sends  voice  data  to  and  from  the 
ACIH  unit 

P 

8 

ACIH  receives  map  data  from  headquarters 

Headquarters  sends  map  layout  of  the  ACIH  position 

P 

It  took  4  tries  for  the  map  layout  to  display  on  the  ACIH 
OLED  screen. 

9 

Headquarters  receives  Thermal  Vision  Video 
from  the  ACIH 

The  ACIH  switches  the  regular  view  to  Thermal  Vision. 
Headquarters  receives  the  new  video  format. 

P 

It  took  a  minute  for  the  user  to  find  the  switch  to  change 
the  display  into  this  mode.  The  user  didn’t  know  how 

10 

Headquarters  receives  Night  Vision  Video 
from  the  ACIH 

The  ACIH  switches  the  regular  view  to  Night  Vision. 
Headquarters  receives  the  new  video  format. 

P 

It  took  a  minute  for  the  user  to  find  the  switch  to  change 
the  display  into  this  mode.  The  user  didn't  know  how 

11 

ACIH  is  able  to  detect  detect  a  person  at 

400  meters 

The  ACIH  detects  a  person  using  zoom  options,  thermal 
and  night  vision  at  a  distance  of  400  meters 

F 

The  ACIH  was  able  to  detect  a  person  under  normal 
vision  using  the  zoom  options  at  400  meters.  But  using 
the  thermal  and  night  vision  was  able  to  detect  a  person 
at  a  max  of  375  meters 
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TEST  SCENARIO  NAME:  ACIH  connection  with  another  ACIH  unit. 


Description: 

Objective: 


Requirement 

Coverage: 

Pass/Fail: 

Comment: 


The  ACIH  (1)  sends/receives  information  to  and  from  ACIH(2). 
Both  units  are  monitored  by  headquarters. 

Verify  connectivity  between  headquarters  and  2  ACIHs 

Verify  connectivity  between  the  two  ACIHs 

Verify  there  is  constant  information  going  back  and  forward 

Verify  video  data  is  received  and  sent 

Verify  text  data  is  received  and  sent 

Verify  voice  data  is  received  and  sent 


ACIH  -  R 1 ,  ACIH  -  R2,  ACIH  -  R3,  ACIH  -  R6,  ACIH-7,  ACIH  -  R8 
PASS 


Step  Requirements 


Description 


P/F 


Headquarters  establishes  a  connection  with 

1  both  ACIH  units  Headquarters  sends  PING  to  ACIH  (1)  and  ACIH(2)  P 


Comment 

ACIH  (2)  took  2  minutes  longer,  the 
user  didn't  know  how  to  turn  on  the 
transmitter. 


Headquarters  receives/sends  voice  data  ACIH  (1)  and  (2)  sends  voice  verification  check  to 

2  from  both  ACIH  units  headquarters 

P 

Headquarters  gets  video  from  both  ACIH 

3  units  ACIH(1)  and  (2)  sends  video  to  headquarters 

P 

Headquarters  gets  GPS  data  from  both 

4  ACIH  units  ACIH(1)  and  (2)  sends  GPS  data  to  headquarters 

P 

ACIH(1)  establishes  voice  communication  ACIH(1)  sends/receive  voice  verification  check  with 

5  with  AICH  (2)  ACIH(2) 

P 

6  Headquarters  sends  map  layout  to  the  ACIH  units  ACIHs  receives  location  map  layout  from  headquarters 

ACIH(1)did  received  the  map  layout, 
but  ACIH(2)  did  not. 
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TEST  SCENARIO  NAME:  Connection  between  10  ACIH  units. 


1 0  ACIHs  units  are  part  of  the  test  and 
are  monitored  and  maintain 

Description:  communications  with  headquarters. 


Objective: 


Requirement 

Pass/Fail: 

Comment: 


Verify  connectivity  between  headquarters  and  the  10  ACIHs 

Verify  connectivity  between  the  10  ACIHs 

Verify  there  is  constant  information  going  back  and  forward 

Verify  video  data  is  received  and  sent 

Verify  text  data  is  received  and  sent 

Verify  voice  data  is  received  and  sent 

ACIH  -  R1,  ACIH  -  R2,  ACIH  -  R3,  ACIH  -  R6,  ACIH-7,  ACIH  -  R8 
PASS 


Step  Requirements  Description  P/F  Comment 


Headquarters  establishes  a  Headquarters  sends  PING  to  3  user  didn't  know  how  to  turn 

1  connection  with  10  ACIH  units.  each  ACIH  P  on  the  transmitter. 


2 

Headquarters  receives/sends 
voice  data  from  both  ACIHs 

Each  ACIH  sends  voice 
verification  check  to 
headquarters 

P 

3 

Headquarters  gets  video  from 
all  ACIH  units 

Each  ACIH  sends  video  to 
headquarters 

P 

4 

Headquarters  gets  GPS  data 
from  each  ACIH  unit 

Each  ACIH  sends  GPS  data  to 
headquarters 

P 

5 

All  ACIH  units  establish  voice 
communication  with  each  other 

Each  ACIH  sends/receive 
voice  verification  check  with 
each  other 

P 

6 

Headquarters  sends  map  layout 
to  the  ACIH  units 

ACIHs  receives  location  map 
layout  from  headquarters 

5  units  did  not  received  the 
map  layout.  Contractor  would 
investigate  this  particular 
ACIHs  units 
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5.  ACIH  DT&E  TEST  REPORT 


DT&E  TEST  REPORT 
For  the 

Advanced  Combat  Integrated  Helmet 


1.  Executive  Summary 

The  following  paragraphs  summarize  the  results  of  the  Advanced  Combat 
Integrated  Helmet  (ACIH)  Developmental  Test  and  Evaluation  (DT&E).  The  purpose  of 
this  test  was  to  assess  the  ability  of  the  Advanced  Combat  Integrated  Helmet  (ACIH)  to 
meet  the  Mission  Critical  Thread:  ACIH  communications  (audio/visual)  shall  not  be 
interrupted  and  compromised  throughout  the  operation. 

2.  Scope 

Testing  was  conducted  at  the  Contractor’s  facility.  The  ACIH  system 
accumulated  72  operating  hours  over  a  7  day  period. 

3.  Background 

The  developmental  test  was  performed  in  preparation  for  the  Operational  Test 
Readiness  Review  (OTRR).  The  testing  included  the  integration  of  the  hardware, 
software,  and  interoperability.  This  evaluation  included  an  assessment  of  the  ACIH 
critical  technical  parameters  in  support  of  the  mission  critical  thread. 

4.  Resources 

Test  resources  include  simulators  (satellite,  desert  environment,  etc.),  GFE  ACIH 
equipment,  bandwidth  limiter,  noise  signal  generator,  wireless  network  sniffer,  wireless 
network  throughput  and  latency  measurement  equipment. 

5.  Test  Results 

The  overall  finding  is  that  the  maturity  of  the  ACIH  during  DT&E  was  found  to  be 
satisfactory.  The  assessment  is  based  on  the  following: 

Critical  Technical  Parameters 

•  The  Advanced  Combat  Integrated  Helmet  (ACIH)  requirements  shall  provide  and 
maintain  audio  and  video  communications  throughout  entire  hostage  rescue 
operational. 

•  The  system  shall  provide  Operational  Awareness  including  the  ability  to  transmit 
and  receive  user  and  team  location,  as  well  as  receive  intelligence  from 
headquarters. 

•  The  ACIH  helmet  shall  share  encrypted  data  to  protect  users  and  the  mission. 

•  The  ACIH  helmet  shall  process  the  data  and  output  the  required  infonnation  to 
the  user. 

•  The  system  shall  have  a  MTBF  of  no  less  than  3900  hours. 
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All  tests  were  performed  according  to  the  Test  and  Evaluation  Master  Plan  (TEMP). 
Some  issues  arose  in  preparation  for  the  test;  contractors  corrected  the  issues  prior  to 
system  testing.  For  system  testing  four  scenarios  were  tested:  satellite  connection  to 
headquarters,  connection  between  endpoints  (headquarters  to/from  satellite  to/from 
ACIH),  ACIH  connection  to  other  ACIH  units,  and  connection  between  10  ACIH  units. 

Some  issues  arose  during  the  test.  Contractors  corrected  issues  before  proceeding.  See 
DT&E  DATA  RESULTS.xls  file  for  DT&E  test  data  and  results. 

6.  System  Assessment 

The  ACIH  was  found  to  meet  all  system  level  requirements  in  support  of  the 
mission  critical  thread.  This  system  level  assessment  is  based  on  extensive  testing  in  the 
laboratory.  Testing  included  end-to-end  perfonnance  in  a  desert  environment  and 
exercised  audio  and  visual  communications. 

7.  Findings 

The  overall  finding  is  that  the  maturity  of  the  ACIH  was  found  to  be  satisfactory. 
Specifics  of  these  findings  include: 

a.  Finding  #1:  Successful  communication  paths  tested:  the  ACIH  successfully 
provided  uninterrupted  audio/visual  communication  throughout. 

b.  Finding  #2:  The  ACIH  met  or  exceeded  all  test  objectives. 

c.  Finding  #3:  Although  communications  were  demonstrated  to  meet  the  test 
objectives,  some  components  lost  communication  for  brief  periods  or  did  not 
receive  all  data. 

Issues: 

a.  Issue  #1 :  The  ACIH  was  only  able  to  receive  4  of  5  GPS  information  artifacts 
from  headquarters. 

b.  Issue  #2:  Only  the  first  ACIH  received  map  layouts  from  headquarters,  the 
second  ACIH  did  not  during  ACIH  to  ACIH  testing. 

c.  Issue  #3:  Only  five  ACIH  units  received  map  layouts,  the  other  five  did  not 
during  the  ACIH  test  with  10  units. 

8.  Recommendations 

Issue  #1  -  #3:  All  issues  pertained  to  loss  of  data.  Although  uninterrupted 
communication  was  achieved,  the  loss  of  data  in  some  components  may  result  in 
interrupted  communication  in  operational  use.  Improve  reliability  of  transportation 
and  receipt  of  data. 
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Other  recommendations:  Address  Integrated  Logistics  Support. 


9.  Conclusion 

ACIH  DT&E  was  successful  in  meeting  its  test  objectives  demonstrating 
hardware,  software,  and  interoperability  maturity.  We  recommend  that  the  ACIH 
proceed  to  OTRR. 
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6.  ACIH  NOTIOAL  INTEGRATED  LOGISTICS  SUPPORT  PLAN 


Logistics 

•  Picatinny  has  been  made  the  ISEA  for  the  ACIH 
-  Will  perform  the  ISEA  role  for  the  helmet 

•  M-Demo  has  passed 


Troubleshooting  and  Diagnostics 

•  Troubleshooting  and  Diagnostics 

-  Troubleshooting  and  diagnostic  equipment  well  be 
shipped  with  the  helmets  to  ensure  working 
helmets 

-  Maintenance  plan  will  be  used 
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Supply 

•  Lowest  Replaceable  Unit 

-  Camera 

-  Microphone 

-  Speakers 

-  Display 

-  Transmitter/Receiver 

-  User  Interface 

•  Spare  Helmets  located  at  HQ 

•  Faulty  helmets  will  be  sent  to  Picitanny 
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APPENDIX  E:  MODEL  DETAIL 


main .m 

%%  Initialize  variables 
tic 


nSys  =  1000;  %  number  of  systems  run 

CTI  min  =  5;  %  minimum  number  of  Critical  Technology  Interfaces 
(CTIs)  a  system  can  have 

CTI  max  =20;  %  maximum  number  of  CTIs  a  system  can  have 

t  =  1;  %  normalized  program  time 

%  define  the  I/ORL  requirement  for  each  review 
MS_pass  =  {}; 

MS_pass.ASR  =  1; 

MS_pass.SRR  =  2; 

MS_pass.SFR  =  3; 

MS_pass.PDR  =  3; 

MS_pass.CDR  =  5; 

MS_pass.IRR  =  6; 

MS_pass.FRR  =  7; 

MS~pass . OTRR  =  7; 

MS_pass.OT  =  7; 

rework  time  =  {};  %  amount  of  additional  time  it  takes  to  fix  a 
I/ORL  problem 

rework^time . ASR  =  .1;  %  note  this  data  was  not  validated,  so  it 
is  not  used  in  the  output 

rework  time.SRR  =  .1;  %  if  actual  data  is  found,  these  values  can 
be  adjusted 

rework^time . SFR  =  .1; 
rework  time.PDR  =  .1; 
rework_time.CDR  =  .1; 
rework  time.IRR  =  .1; 
rework  time.FRR  =  .1; 
rework^time . OTRR  =  .1; 
rework  time.OT  =  .1; 


a  =  2.25; 


scaling  factor 


SE  data 

rework^cost  =  { } ;  %  amount 
I/ORL  problem 

rework^cost . ASR  =  a*. 001; 
rework^cost . SRR  =  a*.002; 
rework^cost . SFR  =  a*.005; 
rework^cost . PDR  =  a*. 0075; 
rework  cost.CDR  =  a*.010; 


to  match  the  relative 
of  additional  cost  it 


costs 

takes 


to  actual 
to  fix  a 
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rework_cost . IRR  =  a*.  015; 
rework^cost . FRR  =  a*.  025; 
rework^cost . OTRR  =  a*.  125; 
rework^cost . OT  =  a*.  25; 

init  =  ones ( 1 ,  nSys )  ;  %  define  an  initial  array  of  ones  with  one 
element  for  each  of  the  number  of  systems 

old  =  struct ( 'pass ', init, ' sched' , init,  cost init,  .. . 

' IORL  avg ' , init, ' IORL  min', init);  %  data  structure  for 
holding  run  info  for  the  old  SE  process 

new  =  old;  %  data  structure  for  holding  run  data  for  the  new  SE 
process 


%%  Run  simulation 

for  i=l : nSys 

%%  initialize  system 

t  =  1;  %  reinitialize  schedule  variable 

new. cost (i)  =  0;  %  initialize  cost  variable 

old . cost ( i )  =  0 ; 

nCTI  =  round (rand* (CTI  max-CTI  min) +CTI_min) ;  %  define  number 
of  CTIs  for  system 

IORL  act  new  =  zeros ( 1 , nCTI ) ;  %  establish  initial  I/ORLs  for 
new  SE  process 

IORL  meas  new  =  zeros ( 1 , nCTI ) ;  %  establish  array  for 
perceived  I/ORL  values 

IORL  act  old  =  IORL  act  new;  %  establish  initial  IORLs  for 
old  SE  process 

%%  ASR 

' ASR' ; 

delta  =  work^IORL ( IORL  act  new,  'ASR') -IORL  act^new;  % 
calculate  actual  I/ORL  improvement  over  the  phase.  this  ensures  that 
the  improvement  is  only  due  to  the  rework  and  not  random  variation. 

IORL  act  old  =  delta  +  IORL  act  old;  %  add  I/ORL  improvement 

to  old 

IORL  act  new  =  delta  +  IORL  act  new;  %  add  the  same 
improvement  to  new 

IORL  meas  new  =  measure  IORL (IORL  act^new, IORL  meas^new) ;  % 
measure  the  I/ORL  levels 
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t  =  t  +  rework  time . ASR*sum ( IORL  meas  new<MS_pass . ASR) ;  %  if 
the  system  doesn't  meet  the  I/ORL  threshold,  add  schedule  based  on  the 
current  review  and  number  of  failured  interfaces 

O, 

o 

this  code  may  need  to  be  adjusted  based  on  the  SE  process  data 
discovered 

new.cost(i)  =  new.cost(i)  + 

rework_cost . ASR*sum ( IORL  meas_new<MS_pass . ASR) ;  %  if  the  system  misses 

the  threshold,  also  add  cost  based  on  the  current  review  and  number  of 
failed  interfaces 

IORL  act  new (IORL  meas  new<MS_pass . ASR)  = 
rework_IORL ( IORL  act_new(IORL  meas_new<MS_pass . ASR) , ' ASR' ,MS_pass) ;  % 
rework  the  I/ORL  value  until  it  passes 

IORL  meas  new  =  measure_IORL ( IORL  act^new, IORL  meas^new) ;  % 
convert  actual  interoperability  values  to  measured  I/ORL  values 

O,  o,  QDD 

o  o  Di\i\ 

' SRR ' ; 

%  rinse  and  repeat  for  the  SRR,  etc. 

delta  =  work^IORL ( IORL  act  new, 'SRR') -IORL  act  new; 

IORL  act  old  =  delta  +  IORL  act  old; 

IORL  act  new  =  delta  +  IORL  act  new; 

IORL  meas  new  =  measure  IORL (IORL  act  new,  IORL  meas  new); 

t  =  t  +  rework  time . SRR* sum ( IORL  meas  new<MS_pass . SRR) ; 

new.cost(i)  =  new.cost(i)  + 
rework_cost . SRR*sum ( IORL  meas_new<MS_pass . SRR) ; 

IORL  act  new (IORL  meas  new<MS_pass . SRR)  = 
rework_IORL ( IORL  act_new(IORL  meas_new<MS_pass . SRR) , ' SRR ' , MS_pass ) ; 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) ; 

%%  SFR 

' SFR' ; 

delta  =  work  IORL (IORL  act  new, 'SFR') -IORL  act^new; 

IORL  act  old  =  delta  +  IORL  act  old; 

IORL  act  new  =  delta  +  IORL  act  new; 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) ; 

t  =  t  +  rework  time . SFR*sum ( IORL  meas_new<MS_pass . SFR) ; 

new.cost(i)  =  new.cost(i)  + 
rework_cost . SFR*sum ( IORL  meas_new<MS_pass . SFR) ; 

IORL  act  new (IORL  meas  new<MS_pass . SFR)  = 
rework_IORL ( IORL  act_new(IORL  meas_new<MS_pass . SFR) , ' SFR ' , MS_pass ) ; 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) ; 

%%  PDR 

' PDR ' ; 

delta  =  work  IORL (IORL  act  new, 'PDR') -IORL  act  new; 
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rework 


rework 


rework 


rework 


rework 


rework 


IORL  act  old  =  delta  +  IORL  act  old; 

IORL  act^new  =  delta  +  IORL  act^new; 

IORL  meas  new  =  measure_IORL ( IORL  act  new, IORL  meas  new) 

t  =  t  +  rework  time . PDR*sum ( IORL  meas  new<MS_pass . PDR) ; 
new.cost(i)  =  new.cost(i)  + 
cost . PDR*sum ( IORL_meas  new<MS_pass . PDR) ; 

IORL  act^new ( IORL_meas_new<MS_pass . PDR)  = 

IORL(IORL  act  new(IORL  meas  new<MS_pass . PDR) , ' PDR ' , MS_pass ) 
IORL  meas_new  =  measure_IORL ( IORL  act  new, IORL^meas  new) 

%%  CDR 

' CDR ' ; 

delta  =  work  IORL (IORL  act  new, 'CDR') -IORL  act  new; 

IORL  act_old  =  delta  +  IORL  act_old; 

IORL  act^new  =  delta  +  IORL  act^new; 

IORL  meas  new  =  measure_IORL ( IORL  act  new, IORL  meas  new) 

t  =  t  +  rework  time . CDR*sum ( IORL  meas  new<MS_pass . CDR) ; 
new.cost(i)  =  new.cost(i)  + 
cost . CDR*sum ( IORL_meas  new<MS_pass . CDR) ; 

IORL  act_new(IORL  meas_new<MS_pass . CDR)  = 

IORL(IORL  act  new(IORL  meas  new<MS_pass . CDR) , ' CDR ' , MS_pass ) 
IORL  meas_new  =  measure_IORL ( IORL  act  new, IORL^meas  new) 

%%  IRR 

' IRR' ; 

delta  =  work_IORL ( IORL  act^new, 'IRR') -IORL  act  new; 

IORL  act  old  =  delta  +  IORL  act  old; 

IORL  act^new  =  delta  +  IORL  act^new; 

IORL  meas  new  =  measure_IORL ( IORL  act  new, IORL  meas  new) 

t  =  t  +  rework  time . IRR*sum ( IORL  meas  new<MS_pass . IRR) ; 
new.cost(i)  =  new.cost(i)  + 
cost . IRR*sum ( IORL  meas  new<MS_pass . IRR) ; 

IORL  act_new(IORL  meas_new<MS_pass . IRR)  = 

IORL(IORL  act  new(IORL  meas  new<MS_pass . IRR) , ' IRR ' , MS_pass ) 
IORL  meas_new  =  measure_IORL ( IORL  act  new, IORL^meas  new) 

%%  FRR 

' FRR ' ; 

delta  =  work  IORL (IORL  act  new, 'FRR') -IORL  act  new; 

IORL  act__old  =  delta  +  IORL  act_old; 

IORL  act  new  =  delta  +  IORL  act  new; 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) 
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t  =  t  +  rework  time . FRR* sum ( IORL  meas_new<MS_pass . FRR) ; 

new.cost(i)  =  new.cost(i)  + 
rework_cost . FRR* sum ( IORL  meas_new<MS_pass . FRR) ; 

IORL  act  new (IORL  meas  new<MS_pass . FRR)  = 
rework  IORL(IORL  act  new(IORL  meas_new<MS_pass . FRR) , ' FRR  , MS_pass ) ; 

IORL  meas  new  =  measure_IORL ( IORL  act  new, IORL  meas^new)  ; 

%%  OTRR 

' OTRR ' ; 

delta  =  work_IORL ( IORL  act  new,  'OTRR') -IORL  act  new; 

IORL  act  old  =  delta  +  IORL  act  old; 

IORL  act  new  =  delta  +  IORL  act  new; 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) ; 

t  =  t  +  rework  time . OTRR* sum ( IORL  meas_new<MS_pass . OTRR)  ; 

new.cost(i)  =  new.cost(i)  + 
rework_cost . OTRR*sum ( IORL  meas_new<MS_pass . OTRR) ; 

IORL  act  new (IORL  meas  new<MS_pass . OTRR)  = 
rework  IORL(IORL  act  new(IORL  meas_new<MS_pass . OTRR) ,' OTRR' , MS_pass ) 

IORL  meas  new  =  measure  IORL (IORL  act  new, IORL  meas  new) ; 


%%  Operational  Test 

%  compare  the  I/ORL  values  to  the  OT  threshold  and  tally 
%  the  number  of  failures 

new. pass (i)  =  sum(IORL  act  new<MS_pass . OT) ; 

%  for  each  failure,  apply  the  OT  rework  costs  for  each 
%  failed  interface 
new.cost(i)  =  new.cost(i)  + 
rework_cost . OT*sum ( IORL  act  new<MS_pass . OT) ; 

%  repeat  for  the  current  SE  process 
old. pass (i)  =  sum ( IORL_act_old<MS_pass . OT) ; 
old.cost(i)  =  old.cost(i)  + 
rework_cost . OT*sum ( IORL  act  old<MS_pass . OT) ; ; 

%%  Data  Logging 

%  store  average  I/ORL  values 

new. IORL  avg(i)  =  mean (IORL  act  new) ; 

old. IORL  avg(i)  =  mean (IORL  act  old); 

%  store  minimum  I/ORL  values 

new. IORL  min(i)  =  min (IORL  act  new) ; 

old. IORL  min(i)  =  min (IORL  act_old) ; 

%  store  schedule  slippage 
new.sched(i)  =  t; 
old.sched(i)  =  1; 

end 
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%  calculate  the  percentage  of  passing  systems 
perc_old  =  sum (old . pass==0 ) /nSys ; 
perc^new  =  sum (new . pass==0 ) /nSys ; 

%  print  data  to  matlab  command  window 
{['(',  int2str(nSys),  '  Samples ) ' ] , ' Old ' , ' New ' ; 'Min 
I/ORL ' , mean (old. IORL  min) , mean (new. IORL  min);'%  OT 

Pass', perc_old, perc_new; 'Cost  Overrun : , mean (old. cost) , mean (new .cost) } 

%  create  the  histogram 
figure ( 1 ) ; elf 
subplot (2,1,1) 

hist ( [old. IORL_min] ,  [0:. 5:9.5]  +  . 25) 
hold  on 

plot ( [MS_pass . OT, MS_pass . OT] ,  [0  l*nSys],  ' r- . ' ) 

axis([0  10  0  l*nSys] ) 

title (' Comparison  of  I/ORL  process  to  Current  SE  Process') 
xlabel (' Minimum  System  I/ORL  Value  -  Current') 
subplot (2,1,2) 

hist ( [new. IORL  min] ,  [0:. 5:9.5]  +  . 25) 
hold  on 

plot ( [MS_pass . OT, MS_pass . OT] ,  [0  l*nSys],  ' r- . ' ) 

axis([0  10  0  l*nSys] ) 

xlabel (' Minimum  System  I/ORL  Value  -  New') 
toe 
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work  IORL.m 


function  CTI_out  =  work_IRL (CTI_in, MS ) 

%  define  mean  and  standard  deviation  of  I/ORL  work 
mean  =  1.115; 
std  =  .215; 

%  determine  the  review  that  the  system  is  being  evaluated 
%  and  apply  the  appropriate  amount  of  I/ORL  work 
switch  MS; 

case { ' ASR ' } 

CTI  out  =  CTI_in+max ( 0 , mean+randn (size (CTI_in) ) *std) 
case { ' SRR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size (CTI  in) ) *std) 
case { ' SFR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size (CTI  in) ) *std) 
case { ' PDR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size (CTI_in) ) *std) 
case { ' CDR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size  (CTI_in) ) *std) 
case { ' IRR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size (CTI_in) ) *std) 
case { ' FRR ' } 

CTI  out  =  CTI_in+max (0, 0+randn (size (CTI  in) ) *std) ; 
case { ' OTRR ' } 

CTI  out  =  CTI_in+max (0, mean+randn (size (CTI  in) ) *std) 

otherwise 

debug  =  'broken' 

CTI_out  =  CTI_in; 

end 


CTI  out  =  min (9, CTI  out) ; 
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rework  IORL.m 


function  CTI_out  =  rework_IRL (CTI_in, MS , MS_pass ) 


%  rework  the  system  until  it  matches  the  requirement  for 

review 

%  if  the  interface  is  close  to  the  requirement,  increase 
small  amount  (.1  units) 


switch  MS; 

case { ' ASR ' } 
CTI_out 
case { ' SRR ' } 
CTI_out 
case { ' SFR ' } 
CTI_out 
case { ' PDR ' } 
CTI_out 
case { ' CDR ' } 
CTI^out 
case { ' IRR ' } 
CTI_out 
case { ' FRR ' } 
CTI_out 
case { ' OTRR ' } 
CTI_out 
case { ' SVR ' } 
CTI_out 
otherwise 
debug  = 
CTI  out 


=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

=  max (CTI 

' broken ' 

=  CTI  in; 


in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 
in+ . 1 , MS 


pass .ASR) ; 
pass . SRR) ; 
pass . SFR) ; 
pass . PDR) ; 
pass . CDR) ; 
pass . IRR) ; 
pass . FRR) ; 
pass . OTRR) 
pass . SVR) ; 


end 


CTI  out  =  min (9, CTI  out); 


the 

it  by  a 
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measure  IORL.m 


function  CTI  out  =  measure  IRL (CTI_in, last) 

%  define  probabilities 

p_c  =  .8;  %  probability  of  getting  the  value  correct 
p  fp  =  .15;  %  probability  of  a  false  positive 
p_fn  =  .05;  %  probability  of  a  false  negative 

p  =  rand (size (CTI  in));  %  create  an  array  of  random  numbers 

%  create  array  to  modify  the  I/ORL  values 
delta  =  zeros (size (p) ) ; 

delta (p<=p_c)  =  0;  %  if  correct,  leave  alone 

delta ( (p>p_c)  &  (p<= (p_c+p_fp) ) )  =  1;  %  if  false  positive,  add 

one 

delta ( (p> (p_c+p_fp) )  &  (p<= (p_c+p_fp+p_fn) ) )  =  -1;  %  if  false 

negative,  subtract  one 

CTI  out  =  min (9, floor (max (last, CTI  in+delta) ) )  ;  %  modify  I/ORL 
values  and  output  the  results 
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